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Executive summary 


The project 


Affordable Maths Tuition is a one to one tutoring programme where pupils receive maths tuition over the 
internet from trained maths graduates in India and Sri Lanka. It is delivered by the organisation Third 
Space Learning (TSL). Tutors and pupils communicate using video calling and a secure virtual 
classroom. Before each session, the pupils’ normal classroom teachers are able to select lessons from 
Third Space Learning’s maths curriculum to target individual learning issues. In this evaluation, the 
tutoring sessions took place once a week, at the same time each week. The intervention was targeted 
at Year 6 pupils who were working at Key Stage 2 level 3 or an insecure KS2 level 4, and was delivered 
over 27 weeks from September 2014 to May 2015 by Third Space Learning (TSL) in an initial testing 
phase, with support from Nesta and Nominet Trust. 


The impact of the intervention was evaluated using a randomised controlled trial design, involving 64 
schools and 600 pupils. Schools were randomised to either receive the intervention or deliver 'business 
as usual’ teaching, which might have involved intensive one to one support for maths. A process 
evaluation was undertaken to understand the perceptions of teachers and pupils, assess whether the 
intervention was delivered as intended, and inform any future development of the intervention. The 
evaluation should be considered an efficacy trial. Efficacy trials aim to test whether the intervention can 
succeed under ideal conditions. 


Key conclusions 


. The impact evaluation found no evidence that the intervention had an impact on Key Stage 2 
maths, compared with ‘business as usual’ teaching and support in Year 6. 


. Teachers were largely positive about the online tuition, and reported that it appeared to improve 
pupils’ comprehension, verbal fluency, and confidence in maths. 


. Schools should consider whether their computer network can support the implementation of an 
online programme. Teachers were positive about the technical support and user experience of 
the programme, but some experienced technical challenges in the implementation. 


. As the online tuition is a ‘talking’ intervention, it appeared to work better when the pupils were 
spaced out in larger rooms so that the noise from other sessions was less distracting. 


. Future research could examine whether the programme has an impact on pupils’ comprehension, 
mathematical capacities, verbal fluency, and confidence in maths, as this was an outcome 
reported by teachers. 


Security rating awarded as 
How secure are the findings? 8 : 
* part of the EEF peer review 


This evaluation had moderate security. It was a well-designed randomised trial, and relatively few pupils 
were lost to the analysis due to issues such as moving school. Two padlocks were removed from the 
rating because the trial was only designed to reliably detect an impact of four months’ progress or more. 
The trial was designed in this way because Affordable Maths Tuition is an intensive intervention, and 
would require a larger impact to be cost-effective. 


What are the findings? 


The impact evaluation found no evidence that the intervention had an impact on the primary outcome 
of the Key Stage 2 maths test, compared with ‘business as usual’ teaching in Year 6. There was also 
no evidence that the intervention had an impact on the Key Stage 2 reading test, or a differential impact 
on pupils who were eligible for free school meals or pupils who took part in more tutoring sessions. The 


Education Endowment Foundation 4 


Affordable Online Maths Tuition 


intervention was generally implemented as intended, so the lack of impact does not appear to result 
from poor fidelity of implementation. The process evaluation suggested several potential explanations 
for the lack of impact observed: 


e Some pupils complained they were interrupted when working as the tutor could not see that 
they were still tackling a problem, or being pushed too hard when they did not understand. This 
may have been due to the lack of face-to-face contact with the tutors. 

e Short-lived technical issues at the beginning of the project. 

e Control group activity, as high-stakes testing takes place in Year 6 and we might expect control 
schools to have employed other forms of intensive tutoring. Five schools in the control group 
reported implementing one to one tuition using face-to-face tutors, and this may have diluted 
the effect slightly. 


The process evaluation also described many positive aspects to the intervention. Schools were largely 
positive about the online tuition, and confident that it was beneficial for their pupils in terms of improved 
comprehension, verbal fluency and confidence in maths. Pupils were also generally positive about the 
impact of the intervention on their own maths capabilities. Teachers commended the programme for its 
clarity and simplicity, good content and objectives linked to the curriculum. Delivery of the intervention 
was well supported by TSL who provided good technical and on-site support. 


This evaluation was undertaken when Third Space Learning was in a relatively early stage of its 
development, and the findings should be considered in this context. TSL is committed to developing the 
programme and has already instituted many of the improvements that this report recommends. 


How much does it cost? 


The programme cost £378 per pupil for 27 weeks of tuition. There was some time required at the 
beginning of the year to set up the intervention. On average, teachers spent 10 minutes per pupil to 
create their academic profile and account. At the beginning of the programme, it took on average 90 
minutes to set up and test the local computers for a class. Each week, it took teachers about 25 minutes 
to select the lesson for the group. Supervising the sessions required 45 minutes of a teaching assistant’s 
time. 


Table 1: Summary table 
Effect Size 


(95% Estimated months’ 
Coxe) ayare(sdalers) progress 
Taltsyave)] 


Security rating Cost rating 


Affordable Maths vs. 
control group 


Affordable Maths vs. 
control group (FSM) 


1 Since this report was published, the conversion from effect size into months of additional progress has been 
slightly revised. If this result was reported using the new conversion, it would be reported as 0 months of additional 
progress rather than -1. See here for more details. 
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Introduction 


The Education Endowment Foundation funded the University of York and Durham University to evaluate 
the Affordable Maths Tuition intervention being delivered by Third Space Learning (TSL) and supported 
by Nesta between 2014 and 2016. 


The intervention aims to help improve pupils’ maths skills while they are in their final year at primary 
school (Year 6), especially the maths skills of pupils who are not making expected progress (defined in 
this trial as working at Key Stage 2 level 3 or an insecure KS2 level 4). In the evaluation, the control 
condition was ‘business as usual’ with a waitlist as the schools were offered the intervention the following 
year. 


In this evaluation, TSL was responsible for implementing and delivering the Affordable Online Maths 
Tuition intervention. They have provided a detailed description of the intervention. 


Intervention 
Summary 


Affordable Maths Tuition is an online one to one numeracy programme designed to support children in 
Key Stage 2. The programme consists of weekly 45-minute one to one maths sessions, conducted 
online and during the school day, with each one to one session targeting the learning gaps that have 
arisen in class. 


Research has shown that one to one learning is one of the most valuable learning interventions, with 
the caveats that it is expensive, hard for schools to organise, and, to achieve its maximum impact, should 
be synchronised with what a child is learning in class. 


To overcome some of these factors, and make one to one tuition available to more children, Third Space 
Learning recruits maths graduates in Asia, training them to be online teachers, and then using 
technology to connect them to children in need in the United Kingdom (UK). This allows TSL to provide 
additional tutoring support that can be integrated into the school timetable via the TSL learning platform. 


Teachers are able to select lessons from TSL’s maths curriculum to target individual learning gaps that 
arise in class, thus supporting their teaching strategies and following best practice. Students then work 
online in secure and engaging virtual classrooms, using headsets to talk to their individual tutor. This 
process encourages discussion and questioning of maths concepts, allowing tutors to diagnose and 
treat problems as they arise in class, while providing immediate feedback to student and teacher on 
progress. 


Overview 


Third Space Learning’s one to one maths programme is specifically designed to support the strategies 
of teachers in primary school, reinforcing their classroom planning. In this evaluation, pupils received 
the invention on a weekly basis for 27 weeks from September 2014 to May 2015. 


Tutors are available from 9am to 4.30pm, Monday to Friday and lessons are booked in advance to 
integrate into the school timetable. Each child works with the same tutor every week, building a 
relationship as they progress through learning gaps that have arisen in class. 


Third Space Learning provides schools with an online account, where teachers create academic profiles 
for the children selected for the programme. These profiles identify the learning and personal 
characteristics of the child with the information made available to tutors. 
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Each week the pupils’ teachers log into TSL to select a lesson for each child. Third Space Learning’s 
curriculum covers all of Key Stage 1 and 2, comprising 274 lessons, allowing teachers to select specific 
concepts to target the learning gaps that arise in class. The teachers’ TSL accounts provide full access 
to the library of lesson plans that are used in the one to one lessons, which can be downloaded for 
planning and offline use. 


The one to one lessons take place on the same day and time each week. Each child works with the 
same tutor, one to one, in a secure virtual classroom. Headsets with a microphone are used to allow the 
tutor and child to talk to one another in live time, discussing a lesson plan that is uploaded on to a shared 
virtual whiteboard (with all the various online tools to answer questions and annotate on the lesson 
content). 


Lesson design is broken down into steps to success, allowing tutors to provide detailed reporting of 
every session for teachers (accessed via their TSL account), both to track progress within a given 
lesson, and to suggest a next lesson based on performance. In addition, tutor feedback tracks 
engagement and effort, ensuring a holistic approach to reporting student progress. 


All tutors are full-time employees who work from TSL’s academic centres in India or Sri Lanka. The 
following safeguarding and training processes are followed: 


e Every tutor is a maths, or maths-based, graduate (e.g. physics, computer science, engineering). 

e Every tutor has a police clearance certificate (PCC), comparable to a UK DBS certificate. 

e All sessions are conducted from TSL’s academic premises, under the management of their 
operational and academic teams (led by UK maths teachers on location). 

e Every session is recorded and stored for safeguarding and professional development purposes. 

e Every week, one of each tutor’s sessions is observed and analysed by TSL’s academic team, 
providing weekly professional development targets in order to constantly optimise teaching 
performance. 

e Tutors have no access to personal student data (email, contact, address). 

e TSL follows all European Union (EU) data requirements. 


There were some differences between the programme as it was implemented in this evaluation, and the 
service that Third Space Learning would normally provide. In this evaluation, schools did not pay the full 
cost of the programme as it was funded by the EEF. Also, Third Space Learning would normally not 
specify the length of time in which a pupil takes part. 


Stage of development 


This evaluation was implemented when Third Space Learning was in an initial testing phase. At the start 
of the evaluation, TSL worked with 70 schools over a 6-month developmental period. Following this 
initial phase, and subsequent to the evaluation, TSL has grown to work with 350 schools, implementing 
many of the product and service improvements that arose during the initial evaluation phase. 


The early stage of development, and subsequent growth and maturity of the service, should be 
considered in the context of these findings. 


Background evidence 


Previous research on one to one tuition indicates that it can be effective (Education Endowment 
Foundation, Teaching and _ Learning’ Toolkit, overview on one to_ one _ tuition: 
https://educationendowmentfoundation.org.uk/evidence/teaching-learning-toolkit/one-to-one-tuition/), 

although the studies included in the synthesis of seven meta-analyses comparing one to one tuition 
showed ‘mixed results’. According to the Toolkit: ‘...In some cases one to one tuition has led to greater 
improvement, while in others tuition in groups of two or three has been equally or even more effective 


Education Endowment Foundation 7 


Affordable Online Maths Tuition 


compared to one to one. Overall, the evidence is consistent and strong, particularly for younger learners 
who are behind their peers in primary schools, and for subjects like reading and mathematics’ (EEF, 
Toolkit). 


Evaluation objectives 


The evaluation aimed to investigate the effectiveness of Affordable Online Maths Tuition on the 
mathematical skills of participating Year 6 pupils struggling with maths compared with Year 6 pupils 
experiencing ‘business as usual’ maths instruction. The primary research question was: 


e What is the effectiveness of the Affordable Online Maths Tuition compared with ‘business as 
usual’ on the maths skills of participating children? 


Secondary objectives included: 


e an assessment of the impact of Affordable Online Maths Tuition on the mathematical skills of 
Year 6 pupils not identified; 

e an assessment of the effectiveness of Affordable Online Maths Tuition on the English skills of 
pupils identified to receive the online maths tuition; 

e anassessment of the impact of Affordable Online Maths Tuition on the English skills of Year 6 
pupils not identified; 

e an assessment of the effectiveness of Affordable Online Maths Tuition on the mathematical 
skills of the subgroup of identified pupils eligible for FSM; and 

e anassessment of the implementation of the project and identification of elements of successful 
delivery. 


Ethical review 


Ethics Committee 

Durham University School of Education Ethics Committee 

York Health Sciences Research Governance Committee (by Chair’s Action) 
Approvals 


Protocol Version 1.3 dated 12/02/2014 Approval given by Durham University School of Education Ethics 
Committee: 11/03/2014 


Protocol Version 1.3 dated 12/02/2014 Approval given by York Health Sciences Research Governance 
Committee (by Chair’s Action): 12/03/2014 


Protocol Version 1.3 dated 12/02/2014 Approval given by Third Space Learning: 06/03/2014 
Protocol Version 1.3 dated 12/02/2014 Approval given by Nesta: 05/03/2014 
Protocol Version 1.3 dated 12/02/2014 Approval given by EEF: 06/03/2014 


Project team 


Sponsor 


Sue Final, Intellectual Property Manager, University of York, Research Innovation Office, Innovation 
Centre, York Science Park, York, YO1O0 5DG. T: 01904 435154F: 01904 435101 E: 
sue.final@york.ac.uk 


Implementation team 
Third Space Learning: Kensington Aldridge Academy, 1 Silchester Rd, London W10 6EX 


Nesta: 1 Plough Place, London, EC4A 1DE. Registered charity in England & Wales 1144091 and 
Scotland SCO042833. 
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Tom Hooper, Founder, Third Space Learning, T: 0203 287 8980 

E: tom.hooper@thirdspacelearning.com 

Oliver Quinlan, Programme Manager: Digital Education, Nesta, T: 020 7438 2500 
E: oliver.quinlan@nesta.org.uk 

Tom Kenyon, Director of Education in a Digital Environment, Nesta, 

T: 020 7438 2500 

E: Tom.Kenyon@nesta.org.uk 


Third Space Learning (TSL) were responsible for intervention implementation, training and delivery, 
including writing a detailed description of the intervention to allow others, if necessary to replicate the 
intervention in other areas (see Appendix S); intervention support throughout the life of the study 


Nesta was responsible for school recruitment (jointly with evaluation team), ongoing relationship with 
schools and liaison with Third Space Learning. 


Trial Registration 


The trial was registered as trial number: ISRCTN54650649 
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Methods 


Trial design 


This trial was a pragmatic cluster randomised controlled trial. A pragmatic design was chosen to reflect 
as closely as possible the implementation of the programme in ‘real life’. Consequently, teachers were 
given significant freedom in choosing eligible pupils for the study as they would do in ‘normal’ teaching 
practice. This enables the results to be generalisable to similar pupils and schools outside the trial. A 
total of 64 schools were randomly allocated to be offered the intervention for their Year 6 pupils either 
in 2014/2015 (intervention group) or in 2015/2016 (acting as the waitlist control group during 2014/2015). 
Teachers at all participating primary schools were asked to identify eight Year 5 pupils, plus three 
reserve pupils towards the end of the academic year who they believed would benefit from online maths 
tutoring in their final year of primary school (Year 6); randomisation was carried out after teachers had 
identified potential pupils. Teachers were encouraged to target pupils who were predicted to achieve 
KS2 level 3 or an insecure KS2 level 4 in maths at the end of Year 6. 


The trial was designed, conducted and is reported to CONSORT standards (Altman et al, 2011) in order 
to minimise all potential threats to internal validity, such as selection bias and a range of post- 
randomisation biases (Cook and Campbell, 1979; Shadish, Cook and Campbell, 2002; Torgerson and 
Torgerson, 2008). In this way, unbiased estimates of the impact of the intervention are provided. The 
pupils in the primary schools randomised to the intervention group received Affordable Online Maths 
Tuition in Year 6 during 2014/2015. 


Eligibility 
Recruitment 


Recruitment of schools preferentially targeted schools with high proportions of pupils eligible for free 
school meals (FSM) and high proportions of children achieving level 3 or an insecure level 4 in maths 
in KS2. In each of four geographical areas (York and Hull, Calderdale, London, Birmingham) we held a 
recruitment event, the purpose of which was to inform schools about the project (including information 
about the intervention, pupil eligibility criteria, data requirements, and design of the evaluation) and to 
invite them to complete an expression of interest form (Appendix B). We used a number of techniques 
to contact schools and invite them to the event including postal invitation, direct email (where possible 
to the headteacher or alternatively a general school email address), Twitter, websites (Nesta, EEF), and 
headteachers’ meetings. All schools interested in attending an event were asked to complete either a 
school attendance proforma (Appendix A) or an online registration form stating which event they would 
be attending. 


School eligibility 


In order for schools to be eligible to take part in the evaluation and to receive the intervention, a primary 
school agreement to participate (Appendix C) was put in place with the schools which specified: 


e enthusiasm for the project and for the teachers’ professional learning; 

e willingness to identify all eligible pupils using pre-specified criteria; 

e provision of school characteristics and baseline data about pupils in Year 5 (in May 2014); 

e willingness to allow random allocation to the Affordable Online Maths Tuition intervention in 
2014 or 2015; 

e willingness to identify ten Year 5 (May 2014) pupils plus three reserve pupils; 

e willingness to implement the intervention throughout the academic year 2014/2015; 

e willingness to implement the intervention only with those identified; 

e agreement to be in the independent evaluation; 
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e willingness to follow the guidance provided by the researchers; 

e provision of a designated space for online tuition sessions for pupils; 
e = areliable internet connection; and 

e = provision of KS1 and KS2 data for all Year 6 pupils (2014/2015). 


Pupil eligibility 


Prior to randomisation, each participating primary school identified eight eligible pupils (plus three 
reserve pupils) using pre-specified criteria; identification took place during the last school term of Year 
5 to enable TSL time to prepare schools for delivery. Where a school could identify fewer than eight 
eligible pupils, all pupils identified were included in the trial and that school did not record any reserve 
pupils. The pre-specified criteria for pupil eligibility to take part in the intervention were: in Year 6 (in 
2014/2015); predicted to achieve level 3 or an insecure level 4 in maths by the end of Key Stage 2 
(based on teacher assessments). Pupils with special educational needs (SEN) were eligible for inclusion 
in the intervention if they met the pre-specified criteria; however, if pupils held a statement for special 
needs they were not eligible for the intervention. 


Schools informed parents of pupils about the study (material provided by the evaluation team). Schools 
sent a list of names, unique pupil numbers (UPNs) and baseline data (including free school meal status) 
for all pupils in Year 6 (in 2014/2015) who did not return an opt-out form to their school. Parents had 
the opportunity to withdraw their child’s data from being used in the evaluation (opt-out consent, see 
Appendix D). The three pupils identified as ‘reserves’ would only receive the intervention in specific 
circumstances, for example if one of the original ten pupils left the primary school permanently or refused 
to use the intervention. Any pupils who were recorded as reserves at the start of the intervention period 
are treated as ‘non-identified’ in analyses as it was not intended that they would receive the intervention. 


Randomisation 


Once pupil baseline data was received, schools were allocated on a 1:1 basis either to receive the 
intervention for the Year 6 cohort in 2014/2015 (the intervention school group) or to receive the 
intervention for the Year 6 cohort in 2015/2016 (the waitlist school control group). The allocation was 
undertaken by the independent evaluation statistician (HB) via minimisation using minimPy (Saghaei 
and Saghaei, 2011) and was conducted in three waves. HB was not involved in the recruitment of 
schools and pupils, ensuring independent randomisation. We chose to use minimisation, rather than 
simple randomisation or stratified randomisation using blocks, because this method of allocation allowed 
a better balance in terms of observable school-level characteristics compared with other allocation 
methods. This improves the credibility and statistical efficiency of a trial wnen the number of schools 
allocated is less than 100. 


Naive minimisation with base probability 1.0 (i.e. deterministic minimisation) was conducted using the 
following as factors: 


e number of pupils on roll (two levels: less than 348 pupils and more than 348 pupils); 

e percentage of pupils eligible for FSM (two levels: less than 29% and more than 29%); and 

e percentage of KS2 maths at level 4 and above in 2012/2013 (two levels: less than 87% and 
more than 87%). 


Cut-off values for levels were chosen based on baseline summary statistics from wave one schools. 
Naive minimisation was deemed to be sufficient as the allocations were conducted in batches, rather 
than prospectively, meaning predictability was not a concern and hence a random element was not 
required. 
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Sample size calculation 


In a previous trial evaluating a one to one maths intervention (Every Child Counts (ECC), Torgerson et 
al, 2011) among primary school children over a single term, an effect size of 0.33 of a standard deviation 
was observed for one to one tuition delivered by a classroom teacher. For the current study, the 
intervention was delivered over nearly three terms; therefore, we might have expected a similar or higher 
estimate. Assuming an effect size of 0.33, an unadjusted intra-cluster correlation of 0.19 (from the ECC 
trial), an identified group size of 10, and a pre- and post-test correlation of 0.67 (from national data, 
Gorard, 2006), approximately 44 schools with 440 children needed to be recruited. Allowing for an 
attrition rate of 15%, we needed around 50-52 schools in our study (i.e. 25 or 26 schools receiving the 
intervention from September 2014) to detect a difference of 0.33 of an effect size with 80% power. 


Funding was available for 60 schools (600 places: 300 identified for intervention and 300 identified for 
a waitlist control). This increased the number of schools which would allow us to detect the effect size 
of 0.33 with 85% power while allowing for 15% attrition. 


Outcome measures 


Key Stage 2 (KS2) standard assessment tests (SATs), which are mandatory national tests, were used 
as the outcome measures in this trial. As schools are required to conduct these tests, missing data 
levels were expected to be low and related to absence or missing papers. The fact that these tests are 
independently marked by individuals external to the schools ensured that markers were blind to 
allocation. Results were obtained from the National Pupil Database (NPD). 


The primary outcome was the KS2 maths SAT fine marked score as defined in the NPD as KS2 maths 
points score using fine grading. KS2 English reading SAT fine marked score was used as a secondary 
outcome. For both of these outcomes, higher scores indicate better performance. The SATs were 
administered as per normal practice within the summer term of the academic year 2014/2015; teachers 
did not have access to the test prior to administration and hence there is no potential risk of bias due to 
‘teaching to the test’. 


The protocol stated that Key Stage 1 (KS1) data was used as the pre-test outcome. It was later decided 
that since KS2 predicted maths scores would be more highly correlated with the primary outcome of 
KS2 maths scores (hence providing more power and precision) these predicted levels and sublevels 
would be used in place of the KS1 data in the primary analysis. The KS2 predicted maths scores were 
collected as part of baseline data collection prior to randomisation. 


Analysis 


A statistical analysis plan (SAP) was developed and is included as Appendix S. This was created post- 
randomisation and prior to receipt of any outcome data. Analyses were conducted in Stata® version 13 
(Stata Corporation, College Station, Texas, USA) and R (R Core Team, 2015) using the principle of 
intention to treat, meaning that all schools and pupils were analysed in the group to which they were 
randomised, irrespective of whether or not they actually received the intervention throughout. All 
‘identified’ pupils were included in relevant analyses regardless of whether or not they actually received 
the intervention. Reserve pupils who were not given an intervention space prior to the intervention 
starting in order to fill places (details given below) were treated as ‘non-identified’ throughout even if 
they later received the intervention. 


Statistical significance was assessed at the 5% level unless otherwise stated. Regression-based 
methods of analysis were used. 95% confidence intervals are provided as appropriate. Model 
diagnostics were used to check model assumptions and transformations considered if assumptions did 
not hold. 
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Effect sizes are presented relating to analyses alongside 95% confidence intervals. In this report, effect 
size is defined as: 


A _— B intervention 
0. +05 


where Bintervention is the difference in mean score between the intervention and control groups, oO: is the 
residual standard deviation, and osis the standard deviation between schools. Such a method (rather 
than simply using Hedges’ g) is required due to the use of multilevel models in analysis. This method of 
calculating the effect size (using total variance) differs from the pre-specified method using the residual 
variance due to changes in analysis guidelines from the funders which were made between production 
of the analysis plan and the report. However, this decision was made prior to any data analysis so was 
not ‘data driven’. Numerical values used to calculate effect sizes for each analysis are presented in 
Appendix R. 


Baseline data 


School characteristics are presented by trial arm to assess balance. Pupil-level characteristics are 
summarised by trial arm both as randomised and as included in the primary analysis for those identified 
to receive the intervention. Information on teacher characteristics and lessons from which pupils are 
intended to withdraw are also summarised by trial arm. No formal statistical testing to assess balance 
was conducted. 


Descriptive analyses 


Raw unadjusted outcome results are summarised by trial arm. An estimate of the intra-cluster correlation 
coefficient (ICC) associated with school for the primary outcome of KS2 maths fine mark score is 
presented alongside a 95% Cl: 


e using data from all pupils attending a participating primary school; and 
e only using data from those included in the primary analysis. 


The correlation between the primary outcome of KS2 maths scores and predicted KS2 level was also 
estimated. Additionally, the correlation between the primary outcome and KS1 maths scores was also 
estimated. 


The lessons from which pupils were intended to be withdrawn summarised. Topics selected for each 
intervention session are presented as frequencies and proportions out of the total number of sessions. 
Audio status for sessions is presented in a similar manner. 


The number of reserve pupils who began receiving the intervention prior to commencement is 
summarised. 


Primary analysis 


The primary objective of this study was to investigate the effectiveness of the intervention on the 
mathematics skills of the identified pupils. The difference in maths attainment between identified pupils 
in the intervention group and those in the control group was compared using a linear mixed model with 
fine mark KS2 maths SAT score as the response variable. Group allocation, FSM status, gender, month 
of birth, and predicted KS2 maths score collected at baseline were included as covariates in the model. 
Adjustment was made for cluster randomisation through the inclusion of school as a random effect. 
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Secondary analyses 


Due to new requirements for analysis by the funding body which came into place following the 
publication of the protocol and writing of the analysis plan, a post hoc repetition of the primary analysis 
adjusting only for predicted KS2 maths scores and minimisation factors of FSM (fixed effect), size of 
school, and proportion of pupils achieving L4+ (both captured in the random effect) was conducted. The 
decision to implement this analysis was taken before the evaluation team had received any outcome 
data. The effect size from this analysis is reported in the executive summary as required by the funder 
despite this not being the pre-specified primary analysis. 


An analogous approach to the primary analysis was employed to compare maths attainment of the 
intervention and control reserve and unidentified pupils to assess for any spillover effects for the 
untreated pupils. 


An analogous approach to the primary analysis was also used to assess for difference between the 
intervention and control identified pupils in terms of the secondary outcome of KS2 English reading 
scores; KS1 reading scores were used as a pre-test measure as no predicted KS2 data were collected. 
This analysis was repeated using reserve and unidentified pupils in the control and intervention groups. 


Subgroup analysis 


The effect of the intervention on identified pupils who are eligible for FSM was assessed via the inclusion 
of an interaction between FSM status and allocation in a repetition of the primary analysis. Statistical 
significance was set at the 10% level as this trial is not powered to detect interactions. 


The funder required the primary analysis to be repeated using data only from pupils eligible for FSM as 
a subgroup analysis. This was conducted, although results should be interpreted in light of the reduction 
in power this approach brings. 


Sensitivity analyses 


It was pre-specified in the SAP (Appendix S) that the primary analysis would be repeated twice with the 
inclusion of: 


e an interaction term between allocated group and whether teaching occurred during or outside 
of school hours to investigate if any effect is linked to additional maths tuition; and 
e maths class as an additional random effect to account for any potential teacher effect. 


However, it was not possible to conduct the first of these analyses due to varying session times within 
schools. 


As it was originally planned that KS1 data would be used as the pre-test for the primary outcome, the 
primary analysis was repeated as an additional sensitivity analysis with adjustment for KS1 maths 
scores rather than predicted KS2 maths scores. 


Compliance 
For the intervention group, compliance was summarised by term and overall in terms of: 


e number and proportion of all sessions where a pupil was on time, late (more than 5 minutes), 
absent and cancelling; 

e number and proportion of all sessions where student engagement was assessed as ready to 
learn/not engaged/very focused/distracted by the tutor; and 

e mean and standard deviation of the end time of the session (negative where session finished 
early and positive where session overran). 
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Non-compliance was summarised in terms of student attendance (attended/absent) with thresholds of 
75% and 50% considered. In addition, the number and proportion of pupils attending 75% and 50% of 
sessions on time were summarised. The impact of non-compliance was assessed using complier 
average causal effect (CACE) analysis taking an instrumental variable approach and accounting for 
clustering using cluster-level means with adjustment for the proportion of pupils with birthdays in each 
month, eligible for FSM, who were female and predicted to achieve each KS2 level. 


Process evaluation 
Design 


The main purpose of the process evaluation was to understand the implementation of the project and 
to identify elements of successful delivery. The findings improve understanding but are not 
generalisable. A cross-sectional design was conducted in three distinct stages. 


Stage 1 of the process evaluation involved visiting and speaking to teachers in schools which had 
previously used Affordable Online Maths Tuition but which were not part of the trial to understand how 
the invention was implemented. The information was collected through informal interviews and 
observations with two pilot schools, and collated into a ‘Top Tips for Schools’ document (Appendix E) 
which was distributed to all intervention schools after randomisation. This initial pilot work took place at 
the end of 2013. 


The purpose of stage 2 of the process evaluation was to identify conditions for success prior to the 
implementation of the intervention. The evaluation team deployed an online questionnaire to all schools 
which had previously used Affordable Online Maths Tuition, the purpose of which was to determine what 
issues schools had experienced and to ask them to suggest ‘top tips’ for those using the programme in 
the future. A key question for this stage was: How could the intervention and the delivery of the 
intervention be improved? These data were then supplemented with information gleaned through 
discussions with the developer and from the two pilot schools at stage 1. In stage 2 data were collected 
in the spring of 2014 and informed the implementation of the intervention at the beginning of the following 
academic year (September 2014). 


Stage 3 of the process evaluation took place during the main trial period (autumn 2014 to summer 2015). 
During this stage, the evaluation team aimed to: 


e understand the perceptions of various stakeholders and whether they saw benefits of the 
intervention; 

e assess whether the intervention was delivered as intended; 

e establish whether any compensatory activities occurred in the control schools; 

e identify the elements of successful delivery; 

e inform appropriate modifications; and 

e understand how the intervention was likely to function if taken to scale. 


Multiple data collection methods were employed, including the following. 


e Case studies. Seven case studies of schools receiving the intervention were undertaken. The 
schools were visited and a total of nine focus groups with 55 pupils receiving the intervention 
were conducted (Appendices M, N, and O), as well as 16 in-depth semi-structured interviews 
with school staff delivering the intervention (Appendices H, K, and L). Three supplemental 
interviews with intervention school staff from two different schools were conducted via the 
telephone. 
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e Surveys: Two surveys with 32 intervention schools (Appendices F and G) and one survey with 
32 control schools (Appendix H) were undertaken. 

e Interviews: Three semi-structured interviews with delivery partners (Appendices K, P, and Q) 
were undertaken. 

e Secondary data: Compliance data were supplied by TSL and analysed by the evaluation team. 


Case studies 


The seven primary schools visited by the process evaluator illustrated a range of typologies, based on 
TSL compliance data (see ‘secondary data’ below for more details): 


e Typology A: None or few problems experienced throughout (2 schools); 
e Typology B: Problems experienced throughout (2 schools); and 
e Typology C: Mixed experiences (3 schools). 


The two additional schools where telephone interviews were conducted represented one each from 
Typology A and Typology B. Each typology was therefore represented by three schools, across nine 
schools in total. 


The schools were located in the centre and suburbs of the cities London, Birmingham, Hull and York, 
and in a town in Cambridgeshire, which illustrated the geographical spread of the trial schools. The first 
school the evaluation team visited was in a pilot study in order to test the effectiveness of the research 
instruments. Since there were no significant changes made to the research protocol this school is 
included in the total of nine visited. 


The data were analysed using key themes which were either predetermined (fitting with the aims of the 
process evaluation), or which emerged from the research: outcomes; implementation; factors 
influencing successful delivery; recommended improvements; and the challenges of up-scaling. The 
analysis identified key trends, similarities, and differences across the data-set. 


Surveys 


Two online surveys were sent to all the 32 intervention schools. The first survey was deployed in January 
2015 and the second at the end of the trial in June 2015. An email with the link to the survey (Qualtrics 
platform) was sent to the email accounts of the nominated staff member at the school for the trial. The 
surveys asked about how the teachers thought the pupils were responding to the intervention, and the 
organization and content of the intervention. Twenty-two (out of the 32 schools) responded to the first 
survey (response rate of 69%) and 19 to the second (response rate of 59%). Two reminders were sent 
via email for both surveys and the surveys were left open for two weeks. 


A short questionnaire was emailed to the 32 control schools to find out if they had complied with the trial 
protocol by not delivering any similar online one to one maths tuition services to their Year 6 students. 
Twenty-six of the 32 schools responded (response rate of 81%). None of them had received online one 
to one maths tuition, but five had delivered one to one face-to-face interventions. 


Interviews 


Two in-person interviews were conducted with TSL staff, which included one with the CEO and founder 
of Third Space Learning and one with an academic manager involved with the trial. A third interview 
was conducted with the programme manager for digital education at Nesta. 


Secondary data 
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Compliance data were collated by TSL and supplied to the evaluation team during the trial. In terms of 
the process evaluation, the compliance data were used as the sampling frame for the selection of the 
case study schools. 


Education Endowment Foundation 17 


Affordable Online Maths Tuition 


Impact evaluation 


Participants 


Baseline data were requested from 73 schools. Nine schools did not provide these data and 
consequently 64 schools (87.7%) were randomised; of these, 32 were allocated to the control condition 
and 32 were allocated to receive the intervention. The number of pupils in the year group of interest in 
these schools had a mean of 49 pupils (SD 21.32) and ranged from 17 to 119. 


It was originally planned that teachers at all participating primary schools would be asked to identify ten 
pupils, plus three reserve pupils, who would benefit from receiving Affordable Online Maths Tuition in 
Year 6. Initial interest in the trial was higher than anticipated and so the number of places allocated to 
each school was reduced to eight in order to allow all schools to participate, while working with a limit 
of 600 funded places (300 of which would be intervention places). Following lack of data return from 
interested schools it was necessary to increase the number of places offered to some schools in order 
to fill the funded places. The order in which schools returned the data to York Trials Unit (YTU) was 
used to decide which schools would be offered up to two additional places (to give a total of up to ten 
funded places) and which schools would continue to be offered eight places. Schools were randomised 
in three waves based on time of return. All schools randomised in the first wave (control and intervention 
schools) were given two additional places; these were filled using the first two reserve pupils the teacher 
had identified (based on the order the reserves appeared in the baseline data provided by schools prior 
to knowledge of allocation). Schools in the second wave were ranked from first returned (1) to last 
returned (n) with higher numbers indicating a later return. Using the ranked list, each wave two school 
was Offered two additional places in turn using the same process as for wave one; this continued until 
all 600 funded places had been filled. 


Reserve pupils used to increase the sample size at this stage (prior to beginning of the intervention 
phase of the trial) are treated as ‘identified’ in the analyses below; any pupils who were reserves when 
the intervention started (regardless of whether they later received the intervention) are treated as ‘non- 
identified’ in analyses. Reserves were selected by schools at the point of baseline data being collected 
and prior to randomisation; therefore, no bias in selection of reserves was expected. Reserves from 
both intervention and control schools were used to increase the sample size with a total of 91 reserves 
across all the schools being used prior to intervention commencement. Following this, schools identified 
a median of ten pupils to receive the intervention with a minimum of five and a maximum of ten. 


Figure 1 shows a CONSORT diagram of participant and school flow through the trial. All randomised 
schools completed the trial and data were sent to be matched with the national pupil database for 3106 
unique pupils from 64 schools. Matching was achieved for 98.6% of pupils (nN=3062), meaning levels of 
missing data were very low. 
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Schools sent postal or email invitation 


(n=552) 


School completing expression of interest 


Did not provide baseline data 
(N=4) 


Number of Y6 pupils in participating 


64 schools (n=3169) 


Randomised (N=64) 


Pupils opting-out of 
data sharing (n=63) 


3106 pupils; 600 identified pupils 


Allocated to intervention (N=32) 
1507 pupils, 300 identified 
Received allocated intervention (N=32) 


Did not received allocated intervention (N=0) 


Lost to followup (N=0) 
Unabie to meaich in NAD n=21 pupils of 1507 
(n=5 identified pupils) 
Discontinued intervention (N=0) 


Analysed (N=32) 
Pupils n=289 of 300 identified pupils 
Excluded from primary analysis (N=0) 
Pupils n=11 excluded due to missing data 
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Allocated to control (N=32) 
1599 pupils, 300 identified 
Received allocated intervention (N=32) 
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Lost to followup (N=0) 
Unabie to meich in NAD n=23 pupils of 15ag9 
(n=4 identified pupils) 
Discontinued intervention (N=0) 


Analysed (N=32) 

Pupils n=289 of 300 identified pupils 
Excluded from primary analysis (N=0) 

Pupils n= excluded due to missing data 
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School characteristics 


Baseline characteristics of the 64 randomised schools are presented in Table 2; characteristics were 
similar in both allocated groups. 


The percentage of pupils achieving level 4 or above at KS2 at baseline was similar between the two 
groups at approximately 85%. The mean school size was around 350 pupils in both groups; however, 
the median school size was smaller in the intervention group than in the control group (279 pupils 
compared with 347 pupils). Both intervention and control groups had very similar percentages of pupils 
eligible for FSM at just over 29%; this is higher than the national average of pupils known to be eligible 
for and claiming free school meals in nursery and state-funded primary schools which was 17.0% in 
January 2014 (Department for Education, 2014). The mean percentage of pupils from minority ethnic 
groups was similar in both groups (40.4% in the control group and 46.0% in the intervention group); 
however, the median percentage was slightly lower in the control group than in the intervention group 
(at 24.2% compared with 32.5%). The national average of pupils of minority ethnic origin in primary 
schools in January 2014 was 29.5% (Department for Education, 2014). The percentage of pupils 
supported by school action plus without a statement of SEN was approximately 10% in the control group 
and 9% in the intervention group. 


Only one school was in special measures at baseline (an intervention school). Two schools had 
previously used an online maths intervention with pupils who were going to be in the target Year 6 group 
(both intervention schools). 


Table 2: Summary of school characteristics 
Control 


fo )V{=) ¢-1| | 
N=64 


Talccyavccvalecey al 
N=32 


N=32 


Percentage of pupils achieving L4 or above at KS2 in 2012/2013 


Mean (SD) 


85.8 (8.88) 


84.2 (13.17) 


85.0 (11.17) 


Med (min, max) 


87.0 (68.4, 100) 


86.0 (41, 100) 


86.5 (41, 100) 


Number of pupils on 


roll in 2013/2014 


Mean (SD) 


354 (133.56) 


344 (128.74) 


349 (130.22) 


Med (min, max) 


347 (143, 700) 


279 (183, 611) 


339 (143, 700) 


Percentage of pupils 


eligible for FSM in 2013/2014 


Mean (SD) 29.3 (17.41) 29.1 (14.86) 29.2 (16.06) 
Med (min, max) 28.5 (5.2, 70) 26 (4.3, 67) 27 (4.3, 70) 
Percentage of pupils from minority ethnic groups in 2013/2014 

Mean (SD) 40.4 (36.36) 46.0 (35.90) 43.2 (35.96) 
Med (min, max) 24.2 (0, 100) 32.5 (2, 100) 31.5 (0, 100) 
Percentage of pupils supported by school action plus in 2013/2014 

Mean (SD) 10.1 (6.59) 8.8 (6.17) 9.4 (6.36) 
Med (min, max) 9.4 (0, 30) 8 (0, 27) 8 (0, 30) 


NB: SD = Standard deviation, Med = median, min = minimum, max = maximum 


Pupil characteristics 


Characteristics of 600 identified pupils in Year 6 from participating primary schools are presented in 
missing data within the NPD). 


Table 3 by trial arm and overall. As this was a cluster randomised trial, randomisation aimed to balance 
the trial arms with regard to cluster level characteristics rather than individual level characteristics. The 
mean age was similar between control arms, as was the proportion of males, proportion of pupils eligible 
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for free school meals (FSM) and proportion in receipt of pupil premium. Pupils were eligible if they were 
predicted a level 3 or insecure level 4 in KS2 maths; proportions of pupils predicted at each level were 
very similar between trial arms. Balance was maintained in the as-analysed sample, where 12 pupils 
were excluded from the primary analysis due to unknown FSM status and 10 pupils were excluded due 
to missing primary outcome data (9 due to lack of matching in the NPD and 1 due to missing data within 
the NPD). 


Table 3: Summary of identified pupil characteristics 


As randomised As analysed in primary analysis 

fexelalige)| Intervention fexelalige)| Intervention 

n=300 n=300 n=289 n=289 
Mean age (SD) 10.4 (0.29) 10.4 (0.28) 10.4 (0.29) 10.4 (0.28) 
Male, n(%) 142 (47.3) 150 (50.0) 135 (46.7) 146 (50.5) 
Current FSM, n(%) 101 (33.7) 87 (29.0) 100 (34.6) 84 (29.1) 
Missing, n(%) T (2.3) 5 (1.7) N/A N/A 
Pupil premium, n(%) 144 (48.0) 128 (46.7) 139 (48.1) 123 (42.6) 
Missing, n(%) 0 (0.0) 4 (1.3) N/A N/A 
Predicted KS2 maths 
level for end of Y6 
Level 3, n(%) 64 (21.3) 65 (21.7) 58 (20.1) 65 (22.5) 
Level 4, n(%) 235 (78.3) 235 (78.3) 230 (79.6) 224 (77.5) 
Level 5, n(%) 1 (0.3) 0 (0.0) 1 (0.4) 0 (0.0) 


Table 4 shows similar characteristics for non-identified and reserve pupils as randomised. As above, it 
should be noted that randomisation aimed to balance the trial arms with regard to cluster level 
characteristics rather than individual level characteristics. The average age was again similar in both 
allocated groups. The proportion of males was similar at around 50% in both allocated groups, as were 
the proportions eligible for FSM and pupil premium. Proportions predicted to achieve each level in KS2 
maths were fairly similar with slightly more control pupils predicted to achieve the expected level 4 
(45.1% compared with 39.9%) and slightly more intervention pupils predicted level 6 (6.9% compared 
with 2.1% in the control group). 


Table 4: Summary of non-identified and reserve pupil characteristics 


Control Intervention 

n=1299 n=1207 
Mean age (SD) 10.5 (0.29) 10.5 (0.29) 
Missing, n(%) 1 (0.08) 0 (0.0) 
Male, n(%) 650 (50.0) 640 (53.0) 
Missing, n(%) 1 (0.08) 0 (0.0) 
Current FSM, n(%) 318 (24.5) 287 (23.8) 
Missing, n(%) 7 (0.5) 19 (1.6) 
Pupil premium, n(%) 456 (35.1) 430 (35.6) 
Missing, n(%) 2 (0.2) 10 (0.8) 
Predicted KS2 maths 
level for end of Y6 
Level 1, n(%) 0 (0.0) 7 (0.6) 
Level 2, n(%) 10 (0.8) 22 (1.8) 
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Level 3, n(%) 76 (5.9) 64 (5.3) 
Level 4, n(%) 586 (45.1) 481 (39.9) 
Level 5, n(%) 598 (46.0) 535 (44.3) 
Level 6, n(%) 27 (2.1) 83 (6.9) 
Missing, n(%) 2 (0.2) 15 (1.2) 


Table 5 shows characteristics of teachers who were due to teach Year 6 pupils in 2014/2015 (the 
implementation year). A total of 61 of the 64 (95.3%) schools provided data on 153 teachers (82 teachers 
from 32 control schools and 71 teachers from 29 intervention schools). Data were provided on an 
average of 2.5 (SD 1.21) teachers per school who would be teaching maths to the Year 6 pupils (min=1, 
max=6). Over one third of teachers were aged 30-39 (36.0%) and more than a quarter were aged 20- 
29 years; there were similar proportions in each arm falling into each category. There was a slightly 
higher proportion of male teachers in the intervention schools, at 25.4% compared with 19.5% in control 
schools. The mean number of years teaching including newly qualified teacher (NQT) year was similar 
in both groups at between 11 and 12 years; the median number of years was 10 with a minimum of 0 
(new NQTs) and a maximum of 37 years. Approximately 12% of teachers were maths specialists. A 
higher proportion of teachers from control schools had a PGCE as their highest qualification than 
teachers from intervention schools (32.9% compared with 15.5%) but intervention schools had a higher 
proportion of teachers with a first degree as their top qualification (69.0% compared with 47.6%). 


Table 5: Teacher characteristics 


Control Intervention Overall 

n= 82 n= 71 n=153 
Teacher age 
20-29 years, n(%) 22 (26.8) 20 (28.2) 42 (27.5) 
30-39 years, n(%) 29 (35.4) 26 (36.6) 55 (36.0) 
40-49 years, n(%) 16 (19.5) 13 (18.3) 29 (19.0) 
50-59 years, n(%) 13 (15.9) 11 (15.5) 24 (15.7) 
60-69 years, n(%) 1 (1.2) 1 (1.4) 2 (1.3) 
70+ years, n(%) 0 (0.0) 0 (0.0) 0 (0.0) 
Missing, n(%) 1 (1.2) 0 (0.0) 1 (0.7) 
Male, n(%) 16 (19.5) 18 (25.4) 34 (22.2) 
Missing, n(%) 1 (1.2) 0 (0.0) 1 (0.7) 
Mean number of years teaching including | 11.8 (8.6) 11.3 (8.4) 11.6 (8.5) 
NQT year, mean(SD) 
Missing, n(%) 3 (3.7) 3 (4.2) 6 (3.9) 
Teacher a maths specialist, n(%) 11 (13.4) 8 (11.3) 19 (12.4) 
Missing, n(%) 4 (4.9) 0 (0.0) 4 (2.6) 
Highest level of qualification 
First degree, n(%) 39 (47.6) 49 (69.0) 88 (57.5) 
PGCE, n(%) 27 (32.9) 11 (15.5) 38 (24.8) 
Other postgraduate degree, n(%) 5 (6.1) 10 (14.1) 15 (9.8) 
Other, n(%) 2 (2.4) 1 (1.4) 3 (2.0) 
Missing, n(%) 9 (11.0) 0 (0.0) 9 (5.9) 
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Outcomes and analysis 
Summary of raw outcomes 


Raw unadjusted mean KS2 scores calculated using fine grading are presented in Table 6 by trial arm 
for those eligible for inclusion in the primary analysis. Mean KS2 maths scores were similar in both 
allocated groups at 25.4 (SD 3.34) in the control group and 25.3 (SD 3.29) in the intervention group. 
KS2 English reading scores were also similar at 26.2 (SD 4.23) and 26.0 (SD 4.45) in the control and 
intervention groups respectively. Proportions of individuals with missing data were low in both cases 
(1.3% and 2.0% in the control and intervention groups respectively). 


Table 6: Raw summary statistics of KS2 scores for identified pupils 
Control Intervention Overall 
n=300 n=300 n=600 


Identified pupils 


KS2 maths scores from fine grading 

Mean (SD) 25.4 (3.34) 25.3 (3.29) 25.3 (3.31) 
Median (Min, Max) 25.8 (15, 32.5) 25.8 (15, 34.9) 25.8 (15, 34.9) 
Missing, n(%) 4 (1.3) 6 (2.0) 10 (1.7) 
KS2 reading scores from fine grading 

Mean (SD) 26.2 (4.23) 26.0 (4.45) 26.1 (4.35) 
Median (Min, Max) 27.2 (15, 33.7) 26.8 (15, 33.7) 26.8 (15, 33.7) 
Missing, n(%) 4 (1.3) 6 (2.0) 10 (1.7) 


Similar raw unadjusted summary statistics are presented in Table 7 by trial arm for non-identified and 
reserve pupils. Mean KS2 maths scores were again similar between the intervention and control groups 
at 29.9 (SD 5.29) and 29.8 (SD 5.06) respectively as were KS2 reading scores at 28.9 (SD 4.26) and 
29.0 (4.11) respectively. 


Table 7: Raw summary statistics of KS2 scores for non-identified and reserve pupils 


Non-identified and Control Intervention Overall 
reserve pupils n=1299 n=1207 n=2506 

KS2 maths scores from fine grading 
Mean (SD) 29.8 (5.06) 29.9 (5.29) 29.8 (5.17) 
Median (Min, Max) 29.8 (3, 39) 29.8 (9, 39) 29.8 (3, 39) 
Missing, n(%) 19 (98.5) 20 (1.7) 2467 (1.6) 
KS2 reading scores from fine grading 
Mean (SD) 29.0 (4.11) 28.9 (4.26) 29.0 (4.18) 
Median (Min, Max) 30 (3, 39) 30 (9, 39) 30 (3, 39) 
Missing, n(%) 19 (98.5) 20 (1.7) 2467 (1.6) 


Descriptive analyses 


Schools were asked at baseline when they envisaged implementing the Affordable Online Maths Tuition 
intervention: during a maths lesson; during another lesson; or outside usual lesson time). Two schools 
(3.1%) did not provide a response, 33 (51.6%) planned to deliver the intervention during another lesson, 
27 (42.2%) planned to deliver outside usual lesson time, and 2 (3.1%) planned to deliver the intervention 
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during a maths lesson. These were both times that were not recommended for the intervention to take 
place. 


The most commonly selected lesson objectives were ‘more number problems in context’ (selected for 
111/6356 sessions, 1.75%) followed by ‘fractions, number lines and simple decimal numbers’ (selected 
101/6356 sessions, 1.59%). Selected lesson objectives were missing for 1116 sessions (17.6%). Audio 
status was missing for 712/6356 sessions (11.2%) but was generally good (4138/6356, 65.1%). 


Intra-cluster correlation coefficients (ICCs) and correlation 


Intra-cluster correlation coefficients (ICCs) were estimated in relation to the primary outcome of KS2 
maths score based on fine grading. Using data from all pupils attending participating schools (n=3057) 
an ICC of 0.09 (95% Cl: 0.06 to 0.13) was found. Using data only from identified pupils (n=590) the ICC 
estimate was higher at 0.28 (95% Cl: 0.18 to 0.38). 


The correlation between the primary outcome of KS2 maths score and predicted KS2 maths level based 
on 3041 pupils was estimated using Spearman’s rho to be 0.67 which was as anticipated in the sample 
size calculation. The correlation between the primary outcome of KS2 maths score and KS1 maths 
scores based on 2921 pupils was estimated using Spearman's rho to be 0.73. 


Regression model results 


Table 8 shows a summary of results for primary and secondary regression analyses. No significant 
differences were found between the two randomised groups in any of the analyses. Results are 
discussed in more detail in the following sections. 


Table 8: Summary of results from primary and secondary analyses 


Score difference* Effect size 
(CEyares)) 

KS2 Maths - identified pupils (primary analysis) | 0.005 (-0.97, 0.98) | 0.002 (-0.31, 0.32) 
Repetition of primary analysis adjusting for pre- -0.06 (-1.03, 0.92) -0.02 (-0.33, 0.30) 
score and minimisation factors 

KS2 Maths - non-identified and reserve pupils -0.50 (1.29, 0.30) -0.13 (-0.37, 0.10) 
KS2 English reading — identified pupils 0.05 (-0.99, 1.09) 0.01 (-0.25, 0.28) 
KS2 English — non-identified and reserve pupils | -0.11 (-0.65, 0.44) -0.03 (-0.23, 0.16) 
* Intervention-Control 


Primary analysis 


The primary analysis was adjusted for baseline predicted KS2 maths level, gender, FSM status, and 
month of birth, and was conducted on identified pupils. After exclusions for missing data relating to any 
of these variables or the response, analysis was conducted on 578 pupils (289 each in the intervention 
and control groups). There was no evidence of a difference in KS2 maths scores calculated using fine 
grading between identified pupils in the allocated groups, with a non-significant increase of 0.005 in 
score for those in the intervention group compared with those in the control group (p=0.99, 95% Cl: - 
0.97 to 0.98). This relates to an effect size of 0.002 (95% CI: -0.31 to 0.32). 
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Secondary analyses 


Repetition of the primary analysis with adjustment for predicted KS2 maths score and minimisation 
factors only 


As detailed in the methods section, a post hoc analysis to comply with new funding body requirements 
was conducted repeating the primary analysis but with adjustment for predicted KS2 maths score and 
minimisation factors of FSM status (fixed effect), size of school and proportion of pupils achieving L4+ 
(both captured in the random effect). This analysis was conducted on the same 578 pupils (289 pupils 
per arm) as those excluded from the primary analysis were missing either primary outcome data or FSM 
status data. There was no evidence of a difference in KS2 maths scores calculated using fine grading 
between identified pupils in the allocated groups, with a non-significant decrease of 0.06 in score for 
those in the intervention group compared with those in the control group (p=0.91, 95% Cl: -1.03 to 0.92). 
This relates to an effect size of -0.02 (95% Cl: -0.33 to 0.30). 


Effect on maths scores of non-identified and reserve pupils 


There was no evidence of a difference in KS2 maths scores between non-identified and reserve pupils 
in the allocated groups, with a non-significant decrease of 0.50 in score for those in the intervention 
group compared with those in the control group (p=0.22, 95% Cl: -1.29 to 0.30). This relates to an effect 
size of -0.13 (95% Cl: -0.37 to 0.10). 


Effect on English reading scores of identified pupils 


There was no evidence of a difference in KS2 English reading scores calculated using fine grading 
between identified pupils in the allocated groups, with a non-significant increase of 0.05 in score for 
those in the intervention group compared with those in the control group (p=0.92, 95% Cl: -0.99 to 1.09). 
This relates to an effect size of 0.01 (95% Cl: -0.25 to 0.28). 


Effect on English scores of non-identified and reserve pupils 


There was no evidence of a difference in KS2 English reading scores calculated using fine grading 
between non-identified and reserve pupils in the allocated groups; with a non-significant decrease of 
0.11 in score for those in the intervention group compared with those in the control group (p=0.70, 95% 
Cl: -0.65 to 0.44). This relates to an effect size of -0.03 (95% Cl: -0.23 to 0.16). 


FSM subgroup analysis 


A pre-specified subgroup analysis included an interaction term between FSM status and trial allocation 
in a repetition of the primary analysis to examine the effect of the intervention of pupils eligible for FSM. 
Statistical significance was assessed at the 10% level. There was no evidence of a statistically 
significant interaction between allocated group and FSM status (p=0.63; 95% Cl: -1.23 to 0.74; ES - 
0.08, 95% Cl: -0.39 to 0.24) suggesting the intervention did not have a differential effect dependent on 
FSM status. As requested by the funder, the primary analysis was repeated using data only from pupils 
eligible for FSM as a subgroup analysis. There was no evidence of a difference between allocation 
groups in KS2 maths between those eligible for FSM randomised to the intervention and control groups 
with a non-significant decrease of 0.16 for pupils in the intervention group when compared with those in 
the control group (p=0.80, 95% Cl: -1.37 to 1.05; ES -0.05, 95% CI: -0.45 to 0.35). Interpretation of this 
result should be made taking into account the reduction of power caused by conducting this type of 
analysis. 
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Sensitivity analyses 


It was planned that a repetition of the primary analysis including an interaction term between allocated 
group and whether teaching occurred during or outside of school hours would be undertaken to 
investigate if any effect was linked to additional maths tuition. It was not possible to conduct this analysis 
due to the delivery time of sessions varying within school. 


The second planned sensitivity analysis repeated the primary analysis including an additional random 
effect to account for any potential teacher effect. This analysis was conducted on 244 pupils for whom 
maths teacher information was provided, out of the 600 total pupils identified. There was no evidence 
of a difference between identified pupils in each allocated group with a non-significant decrease of -0.39 
for intervention pupils compared with control pupils (p=0.57, 95% Cl: -1.74 to 0.96) when accounting for 
maths teacher. Interpretation of this result should take into account that this analysis was conducted on 
fewer than half of participants (41% if 600 pupils) from fewer than half of the schools (42%, 27 of the 64 
schools) and that schools providing teacher data may differ from those who did not provide these data. 


When the primary analysis was repeated adjusting for KS1 maths scores, results were consistent with 
the primary analysis. After exclusions for missing data relating to adjustment or response variables, 
analysis was conducted on 547 pupils (272 the intervention group and 275 in the control group). There 
was no evidence of a difference between the allocated groups in terms of KS2 maths scores with a non- 
significant increase of 0.03 in score for those in the intervention group compared with those in the control 
group (p=0.94, 95% Cl: -0.86 to 0.92). 


Compliance 


Compliance data were available in relation to 308 pupils from the 32 intervention schools. Pupils had an 
average of 20.6 (SD 4.73) entries, each relating to one session, with a minimum of 1 entry and a 
maximum of 29 entries (median 21). Data were provided on 285 identified pupils, 13 reserve pupils, and 
10 pupils who were not originally identified nor identified to be reserve pupils. 


Table 9 shows attendance data both overall and by term; data were available for 5834 sessions (91.8%). 
Overall, pupils were on time for 52.7% of sessions (n=3347); more than 5 minutes late for 36.2% of 
sessions (n=2303); absent in 132 cases (2.1%); and the session was cancelled in 52 cases (0.8%). 
Absence was higher in Term 1 than in Terms 2 and 3 (at 4.1% compared with 0.6% and 0.7% 
respectively). 


Of the sessions cancelled, over one quarter were scheduled for 2pm (49/184, 26.6%); a further 15.8% 
were scheduled for 1pm (n=29); 13.0% were scheduled for 3:30pm; and 10.3% were scheduled for 
2:30pm (n=19). 


Table 9: Attendance summary 


n=2655 n=2823 n=878 n=6356 
On time, n(%) 1228 (46.3) 1562 (55.3) 557 (63.4) 3347 (52.7) 
Late, n(%) 756 (28.5) 1242 (44.0) 305 (34.7) 2303 (36.2) 
Absent, n(%) 110 (4.1) 16 (0.6) 6 (0.7) 132 (2.1) 
Cancelling, n(%) 41 (1.5) 1 (<0.1) 10 (1.1) 52 (0.8) 
Missing, n(%) 520 (19.6) 2 (0.1) 0 (0.0) 522 (8.2) 
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Student engagement (as assessed by the tutor) is summarised in Table 10, both overall and by term; 
data were missing in relation to 21.6% of sessions (n=1373). For over 60% of sessions students were 
either ‘very focused’ (22.3%, 1417 sessions) or were ‘ready to learn’ (46.8%, 2975 sessions). 


Table 10: Student engagement 


Term 1 Term 2 Term 3 Overall 
n=2655 n=2823 n=878 n=6356 
Very focused, n(%) 364 (13.7) 813 (28.8) 240 (27.3) 1417 (22.3) 
Ready to learn, n(%) 818 (30.8) 1663 (58.9) 494 (56.3) 2975 (46.8) 
Not engaged, n(%) 104 (3.9) 219 (7.8) 80 (9.1) 403 (6.3) 
Distracted, n(%) 32 (1.2) 108 (3.8) 48 (5.5) 188 (3.0) 
Missing, n(%) 1337 (50.4) 20 (0.7) 16 (1.8) 1373 (21.6) 


Session end time was recorded in 4987 cases (78.5%) and was recorded so that 0 represented a 
session which finished on time, a positive session end time indicated overrunning by the specified 
number of minutes, and a negative end time indicated that the session finished early by the specified 
number of minutes. Just under 50% of sessions finished on time (n=3058, 48.1%). Of the 6356 sessions, 
1685 (26.5%) overran and 244 (3.8%) finished early. The mean session end time was 0.7 (SD 2.3), 
indicating that sessions on average finished on time (overrunning by approx. 42 seconds). The shortest 
session ended 30 minutes prior to the planned end time and the longest session overran by 15 minutes. 


CACE analysis 


To calculate the number of sessions attended it was assumed that pupils were not in attendance where 
attendance data were missing, in order to be conservative. 


Of the 308 individuals with attendance data, 89.0% (n=274) attended at least 50% of the 27 sessions 
(i.e. attended at least 14 sessions); and 29.9% (n=92) attended at 75% of sessions or more (i.e. attended 
at least 21/27 sessions). The mean number of sessions attended was between 18 and 19 sessions 
(18.3 sessions, SD 4.49) with a median of 19 sessions (min=1, max=27). 


When non-compliance was accounted for through CACE analysis applying a 50% compliance cut-off, 
results were consistent with the primary analysis with no evidence of a difference between the allocated 
groups (p=0.75) and a non-significant increase of 0.02 in mean KS2 maths scores based on fine grading 
for those in the intervention group compared with those in the control group (95% Cl: -0.11 to 0.15). A 
similar result was seen when a 75% compliance cut off was applied, with no evidence of a difference 
between the allocated groups (p=0.74) and a non-significant increase of 0.05 in mean KS2 maths scores 
based on fine grading for those in the intervention group compared with those in the control group (95% 
Cl: -0.27 to 0.38). 


Ancillary analyses 


As requested by the funding body, a calculation of the actual minimum detectable effect size calculated 
was conducted using the observed correlation of 0.67 (as assumed in sample size calculation), the 
observed adjusted ICC of 0.28 in identified pupils (as opposed to the assumed 0.19), the number of 
individuals included in the primary analysis (289 per arm), and the average cluster size of 10. Applying 
the above assumptions gives an effective sample size of approximately 150 pupils meaning that for this 
trial we had the ability to detect an effect size of 0.33 with 80% power. 
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Process evaluation 


This section includes the results of the process evaluation using case study, survey, and interview data. 
Implementation 
Space 


The delivery of the intervention occurred in a variety of different rooms across the schools, most 
commonly in ICT suites, libraries, and spare classrooms, community and training rooms. Typically, the 
pupils were all together in the same place on their own. On the occasions when other pupils were present 
this could be distracting. Some staff liked the ICT room because the computers were already set up, 
although this was an area that sometimes other pupils entered, interrupting pupils’ concentration. As the 
online tuition is a ‘talking’ intervention, it worked better when the pupils were spaced out in larger rooms, 
as this reduced noise crossover between each pupil’s session. Having an appropriate space for the 
intervention was stressed by TSL to all participating schools. The tutors sometimes commented to the 
children about this issue and this was found to be distracting for the children as they could do little about 
it. 


Length and timing 


Most staff and children felt the 45-minute sessions were appropriate in terms of duration, although a few 
children found it difficult to concentrate one to one for this length of time. Staff were divided as to whether 
27 sessions over 3 terms was the optimum length of time. Some thought this length of time enabled 
learning to become embedded, but others felt shorter time-spans and/or greater frequency would be 
more effective. There is more flexibility with time-span outside of the trial conditions. Afternoon sessions 
were the best time for schools as core subjects were delivered in the morning. However, one school 
delivering the intervention on Friday afternoons said the children were tired, and another delivering the 
intervention after school on Fridays experienced the highest pupil drop-outs. The children complained 
about what they missed, which tended to be topic classes, with the strongest complaints from a group 
of pupils who missed PE. It worked well when the other children also did extra maths, as the intervention 
children did not feel they were missing out. Staff noted that missing other classes was no different from 
missing classes due to other interventions, although the duration of three terms for this intervention was 
longer than usual. 


Target group 


Generally staff felt the intervention worked well for Year 6 students, although several recommended 
using it in earlier years, such as straddling Years 5/6, or in Year 4 or 5, or to address the children’s gaps 
and problems before they became too embedded. 


Technical 
Initial set-up 


The biggest barrier to a smooth implementation of the online sessions was technology problems. As 
one teacher said, ‘online it’s only ever as good as your connection and your equipment’. In September 
2014, Third Space Learning (TSL) experienced initial challenges with the set-up due to problems with 
the internet platform, resulting in a delayed start across all schools. By the end of October set-up was 
in place everywhere but some schools continued to experience issues. Of the schools that experienced 
difficulties, the severity of the problems varied, with some schools reporting only intermittent issues, 
while others experienced severe problems throughout. The most common problems were with the 
internet and audio connections, but other issues included difficulties logging on, delays, sound 
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interference, and problems with school systems, drivers, service providers, old machines, microphones, 
headphones, security lock-out, and children clicking on tabs and losing everything. 


On-going technical issues 


Setting up for the online sessions could be lengthy and stressful, with the time spent varying between 
ten minutes and one and a half hours. One member of staff complained of having to install constant and 
numerous programme updates, with the three-minute pre-session warning not being long enough. The 
overall effects of the technological problems were inability to access the programme effectively, time- 
wasting, and stress for both staff and children, which impacted on outcomes as ‘when it doesn’t work 
immediately children get frustrated really fast’. 


Online delivery 


One complaint from the children connected to the online mode of delivery was that, because the tutors 
could not see the children, they often interrupted their concentration by asking what they were doing, 


Are you done? Are you done? ... that gets me annoyed because | was doing my work... 
sometimes | took the headphones off to calm down. 


Some children reported feeling rushed and stressed. The children complained of background noise, at 
the tutor end and hearing pupil-tutor exchanges. However, most schools reported that the sessions 
overall operated sufficiently well, with some describing set-up and operation as straightforward and 
stress-free. When the sound failed the pupils could communicate by typing into a chat box, and on the 
whole TSL technical support was highly rated (21 out of the 22 schools who responded to the January 
survey) and considered accessible, including site visits to sort out problems. Even when there were 
issues, most staff felt it was worth struggling with them because of what the pupils gained from the 
intervention. 


Human investment 


The key staff roles required to deliver the intervention were lead contact for TSL, setting the learning 
objectives (LOs), and the session administrator (SA). 


Lead contact 


The lead contact was also often responsible for the initial set-up of the intervention, involving several 
hours’ work such as collating pupil data, selecting pupils, and setting up their profiles. However, most 
schools which responded to the first survey in January reported that the setting up of the TSL account 
(18 out of 22 schools) and student profiles (15 out of 22 schools) was easy. 


Setting learning objectives 


Many schools conducted their own ongoing monitoring of the participating pupils throughout. Setting the 
pupils’ LOs took between 15-30 minutes per week. Staff generally found this straightforward, and 
commented that the TSL website was user-friendly. Schools reported in the January survey that the TSL 
LOs matched up with their students’ own individual LOs for all (6 out of 22 schools) or most students 
participating (15 out of 22 schools). 


When the Year 6 teacher set the LOs they often tried to make connections between the classroom 
learning and tutor sessions. However, given the weekly nature of tutor sessions and the need to set the 
LOs in advance, it was not always easy to achieve a smooth link. So approaches varied and sometimes 
staff concentrated on filling individualised learning gaps of embedding foundational maths topics 
instead. TSL recommends the teacher set the LOs but often they relied on the tutor suggestions, 
because this was quicker, easier, and/or they trusted the tutor suggestions. 
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Although the Year 6 teacher was academically well positioned to set appropriate LOs, they already had 
very heavy workloads, so sometimes senior managers, such as a maths co-ordinator or data-manager, 
stepped in. Year 6 teachers were often involved in initially selecting the intervention children and setting 
their profiles, but as they did not always know the children well at his stage, it was more effective and 
efficient if these tasks were done by staff who did. 


Session administrator 


SAs monitored the children during the 45-minute intervention, and often set up the computers before 
they came in, which took between 10 and 90 minutes. Each school organised these roles differently. 
Sometimes a senior manager did everything, or sometimes the tasks were distributed between several 
people, such as the Year 6 teacher and a teaching assistant (TA). 


Time variation 


There was variation in the amount of time invested by the staff. When several staff members covered 
the different tasks it spread the workload, although ensuring there was adequate communication 
between them was challenging. The TAs often did the SA role. They were in a good position to support 
the children’s learning and confidence, and observing the sessions helped them make links between 
tutor and classroom lessons. Sometimes a senior manager did everything and this seemed effective in 
terms of continuity, workload and commitment, working especially well if they taught some maths to the 
children. Generally, the support and enthusiasm of the senior leadership team helped drive and underpin 
children and staff engagement and successful delivery of the intervention. 


Most staff were willing to invest their time and labour because they felt the outcomes for the children 
made it worthwhile. Even the SA who spent the most time setting up the computers before each session 
began (up to 90 minute) and found it very stressful, said, ‘I think the results, what the children get out of 
it really, more than overcomes what time it took me to do that’. 


Support and communication 
Provider-school 


TSL say that ‘the success of the intervention is built on establishing good relationships with the schools, 
in order to understand their needs and help them get the most out of Third Space’. Most staff considered 
the customer support from TSL to be good or excellent, with TSL regarded as helpful, approachable and 
responsive. Staff appreciated the technical support, school visits, having a specific TSL person assigned 
to them, TSL talking to the children, and the fact that the TSL advisor was a former teacher. 


However, there was some room for improvement. For example, when one school was promised a child 
could have their previous tutor back this did not happen and the school were not informed, resulting in 
the child dropping out. One SA reported a ‘we are right and you are wrong’ attitude from TSL when they 
needed technical support, and phone calls from TSL about children’s bad behaviour were not always 
balanced with positive feedback. Occasionally the communications from TSL were felt to be a bit 
‘markety’, and several staff felt they got too much contact from TSL. They sometimes complained that 
staff not directly involved in the intervention were copied into emails, even after TSL were requested to 
remove them. Reminders about the LOs were not always well received, especially when received before 
the 24-hour in advance period. However, TSL worked to improve communication with schools during 
the trial year. They recognised that different schools wanted different amounts of contact, and tried to 
provide a more sensitive tailored service, introducing formal agreed contact times: ‘We’re booking in a 
time to arrange a call every few months or every few weeks with the teacher, so it's a set time in the 
schedule... as opposed to these sporadic calls and sporadic communication’. 


Staff-student 
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The communication between the school staff and children also had an impact, with more positive 
outcomes reported where the children had the chance to feed back and talk to staff about the 
intervention, with staff enthusiasm also driving pupil engagement. How staff ‘sold’ the intervention was 
important, working best when it was presented as something special that the children had been chosen 
for because of their good maths work, rather than saying the children needed help with their levels, 
which could re-enforce feelings of inadequacy. TSL are aware that ‘there’s a student sell, because if 
they don’t know what they’re getting out of it they won’t be as invested’. Therefore, they have improved 
the children’s introduction to the intervention by coming in to schools to talk to the children and show 
them a video about what to expect. 


Fidelity 
Intervention schools 


The intervention was implemented as intended, and fidelity to the trial conditions was high. Pupil 
selection was in accordance with the trial protocol, as all the pupils selected were at risk of not achieving 
level 4 in their SATs. Their starting levels varied between level 1 and level 3B. 


The TSL compliance data indicates that for some of the schools punctuality was a problem but 
sometimes start times were delayed because of problems logging in. Occasionally schools missed 
sessions, including due to delayed starts because of technical problems. Some, but not all, were made 
up by extra weekly sessions. A few children had high absences, but this is not regarded as different 
from the control schools (i.e., business as usual). In most schools the same children did the intervention 
throughout, with only a few substitutions. 


Most schools simultaneously ran other maths interventions for Year 6 pupils leading up to the SATs. In 
two cases schools ran one to one tuition for shorter durations face-to-face, and there was one report of 
a child having one to one tuition outside school hours. But there were no other one to one online 
interventions. As intended, the intervention did not replace maths lessons, although one intervention 
group regularly missed 15 minutes of their classroom maths lesson, and in another instance the non- 
intervention children received extra maths at the same time the Affordable Online Maths Tuition 
intervention ran. 


TSL noted there were differences between the implementation in the trial schools and in the usual paying 
schools. TSL did not have as much time as usual to set up with the schools pre-intervention. 
Furthermore, the schools’ initial contact was with the research team, rather than with TSL, eroding TSL’s 
capacity to form good relationships with schools, which TSL regards as the foundation for success. TSL 
reported that poorer initial relationships with schools meant that staff were less likely to deal well with 
problems, which was borne out in the TSL compliance data indicating a poorer user experience. Where 
TSL had been able to conduct early orientations with schools ‘they got engaged and communication 
was good throughout’. 


Control schools 


The fidelity of the control schools to the trial conditions was considered to be high. In June 2015 the 32 
control schools were sent a short questionnaire asking for details of extra maths tuition delivered to 
participating control students between September 2014 and the May 2015 SATs (Appendix H). Twenty- 
six schools responded: five reported pupils receiving one to one tuition in a face-to-face context, but 
none via an online tuition intervention. Given that one to one tuition/support/help is commonplace in 
schools, then we considered this to be ‘business as usual’, although it could be argued that, for these 
five schools, some of the pupils received a partial intervention. These schools were clear, however, that 
they did not use an online one to one real-time intervention. While not conclusively known, it did not 
seem apparent that any alternative one to one tuition was delivered year-long. Three schools reported 
pupils engaging with online maths packages, but none involving live tuition via a teacher. In talking to 
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staff about these types of packages, it became clear that they are very different from the TSL intervention 
in that they were not ‘live’ one to one tuition. In sum, as reported by the schools, none of the pupils 
received one to one online tuition of a type similar to that delivered by TSL. 


Outcomes 
Attainment 


Staff at most schools felt that the pupils had improved attainment beyond the usual expectations due to 
the intervention (15 out of the 19 schools who responded to the second survey), with several noting that 
the children had made good or excellent progress. For example, one teacher said 


The programme was really positive in terms of the outcomes... | think it’s helped dramatically... 
and | think the SATs results will reflect that because | think they've jumped levels. 


While the teachers were cautious about the difficulty of separating out the Affordable Online Maths 
Tuition intervention from other teaching impacts, the general opinion was that the Affordable Online 
Maths Tuition intervention had played a part in the children’s improved achievement levels, which in 
some cases had been dramatic. One teacher said 


When it came to the SATs she was dying to do it, basically wanted to go in there. | said to her 
are you ok today for maths? She said ‘fine why wouldn't | be? | know what I’m doing, I’m going 
to go and show you’, which is a massive change from what she was. | know you can’t isolate a 
single factor, I'm sure the teacher as well, but | think it’s certainly made a difference. 


Achievement was linked to the one to one delivery described by one teacher as ‘Ideal for the kids to 
help levels go up’. Most schools reported (in the second survey) that there were no disadvantages to 
the intervention being one to one (13 out of the 22 schools that responded). 


Most of the children reiterated the perspective that the intervention had helped their maths. They cited 
a positive impact on their SATs, in relation to specific topics, use of methods, and familiarity with the 
questions. One child said ‘I did better in my SATs because of it... because | knew more things’. 


Capacities and comprehension 


Most of the staff thought the intervention improved mathematical capacities and comprehension. They 
said the tuition helped the children work faster, embedded learning, and elicited ‘light bulb’ moments of 
sudden understanding. The SAs observed learning during the sessions. The teachers commended the 
programme for its clarity and simplicity, good content and objectives linked to the curriculum, the set out 
of the methods and teaching of written calculations. 


There was positive feedback on the one to one nature of the intervention, such as the provision of 
bespoke teaching with precise focus on individualised gaps, children being able to work at their own 
speed, the time they were given and the constant dialogue with immediate feedback. One teacher said, 
‘It's what's needed. Brilliant ... a massive advantage’, and the children also appreciated it, 


(it) made me feel very happy that I’ve got someone there beside me to help me get through it. 
And like with the teachers in class are all rushing about to help other people. So if you're stuck 
you can ask her for help and she'll just jump on it straight away and help you. 


Most of the children reported that the tutoring helped their mathematical capacities and understanding, 
and provided them with useable methods. They appreciated the step-by-step guidance with different 
ways into a topic, such as use of visual aids and diagrams, the tutors explaining and modelling examples, 
and the children having to explain their thinking. 
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Both children and staff said the learning filtered into the classroom as children recognised mathematical 
problems, modelled different methods to other students, and applied mental arithmetic. Occasionally 
staff and children reported confusion when the tutor introduced a new method, but mostly the methods 
taught were seen to complement, consolidate or expand classroom learning. 


While some staff commented that the intervention deepened the children’s understanding, others felt 
the teaching could be formulaic and shallow, unable to fully engage with the children’s individual learning 
styles, which is possible with face-to-face teaching. However, this did not negate the value ascribed to 
the intervention for embedding foundational maths knowledge and capacities, plugging specific gaps, 
and modelling methods. 


Verbal fluency 


The intervention is focused on talking through the maths, which TSL regards as one of best ways for 
children to learn: ‘children talking their thoughts and problems you uncover a new set of misconceptions 
that you would never have thought about in the classroom’. 


Verbal reasoning 


The staff valued the opportunity for the children to improve their verbal reasoning and mathematical 
vocabulary, which was particularly useful for word problems in which ‘they can do the maths but can’t 
get the maths out of a contextual word problem’. Staff said having to explain improved the children’s 
maths comprehension, which was reiterated by some of the pupils; for example, one said ‘(talking) made 
me think about it more, but | think I’ve done good’. There were reports of improved verbal fluency and 
communication skills in class, with children who had previously not wanted to verbally contribute being 
more willing to do so. 


The SAs observed a lot of talking during the sessions and Nesta, who also observed a session, reported 


What impressed me the most was the depth of discussion that the children were having with 
the tutors. They were being asked to talk through their methods in a very detailed way. In a way, 
from my experience of teaching in primary, you would find it very hard to do with a significant 
number of children during a normal lesson. 


Nesta’s Programme Manager reported that the children were encouraged to verbalise and delve into 
their understandings and misconceptions, so the interaction was not one-sided and didactic, but 
dialogical and conversational. 


Differences by students 


Staff gave mixed reports as to which students found the talking aspect of the intervention most useful. 
Some staff thought it suited the more confident and talkative children, but others felt the quieter children 
benefited because in the one to one situation they had to speak. Similarly, one teacher thought the 
pupils with English as an additional language (EAL) benefited because of the opportunity to engage 
with, and articulate, worded mathematical problems, while another teacher said the verbal aspect of the 
intervention was a barrier for her EAL students. 


Some children found explaining the maths challenging, but most seemed to enjoy talking, for example 
saying 


It was good ‘cause (in class) you'd get other people answering for you. 
What really helps me is the tutor, they give you a chance to speak. 


Confidence 
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Self-confidence 


Affordable Online Maths Tuition aims to improve results by building confidence in maths, and both the 
staff and children reported increased confidence growing alongside improved abilities. For example, of 
the schools that responded to the second survey, two-thirds reported that the intervention had been 
effective in raising student self-confidence and children said 


It makes you like you can do more. It makes you feel confident. 


The relationship between approval, confidence and achievement, central to Affordable Online Maths 
Tuition’s holistic approach to learning, is demonstrated here. Some staff reported a transformation in 
the children’s improved confidence. One teacher said 


! really like the programme... because | think a lot of the children went on to make progress 
because of the confidence built up... | would say that was a very important lesson from the 
programme. 


Confidence contributed to children’s greater willingness to contribute in the classroom and ‘have a go’. 
They found it easier to say what they found difficult and ask for help. One child said 


When | first started maths | wouldn’t answer a question, I'd just sit there and not put my hand 
up. But now I’m just sticking my hand up, sticking my hand up, trying to get the answer right. 
Even if 1 don’t get it right | aren't really bothered, at least I’ve tried. 


The findings also affirm Affordable Online Maths Tuition’s profile of the type of child that the intervention 
works most effectively for as 


a child whose failure is due to lack of confidence, so they can do it, they just don’t think they 
can, therefore they don't engage. 


The points and rewards were valued because they confirmed to the children that they were able. Many 
of the children were more confident about assessments, and felt better prepared for secondary school. 
One said, 


! used to be not that confident with multiplication and then the tutor explained it to me ... and 
when it came to SATs and there were lots of multiplication questions | were like confident... and 
my teacher looked at the paper and said you're really good. 


This then fits with the Affordable Online Maths Tuition ethos of raising achievement through support and 
encouragement. 


Most staff and children reported that the rewards, in the form of pictures, games, and points, were 
motivational, although a few children complained of gender/age/culture inappropriate images, or that 
the reward system was unfair. Several children engaged more because they felt less exposed than in 
class. They said that if they made a mistake ‘no-one laughs’. One child said they felt ‘yes more confident, 
and | like (maths) now. | didn’t used to like it’. 


Enjoyment and engagement 


It has been indicated then that increased confidence leads to increased engagement, and both staff and 
children gave positive feedback in this respect. 


Staff reports 
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The SAs unanimously reported that the children were well engaged during the sessions, as they 
observed them talking, asking questions, and concentrating throughout. Several staff felt the novelty of 
learning on the computer facilitated the children’s interest, with one teacher saying 


we've loved it. The kids have got lots out of it. They've enjoyed it. They have been engaged in 
their maths, and in a different way. Not with a teacher standing at the front droning on... it 
engages them because they're on a computer... it’s a treat. 


Student reports 


Overall the children’s responses were mixed, with some saying the intervention was interesting and fun, 
but others making negative comments such as ‘it was really bad’, ‘I hated it’ and, it was ‘very boring’. 
Boredom due to repetition was a key factor in disengagement, although some children appreciated the 
repetition, ‘because | was learning the same thing every week | got better at it and quicker at it’, for 
others the repetitious process was off-putting 


whenever you learn something and then they tell you again how you learned, and then they tell 
you again, and then they tell you again, and that’s boring, and again and again. 


Repetitiveness 


The children also complained of doing exactly the same lesson in consecutive sessions: ‘it was ok doing 
the same subject but we kept on doing the same sheet’, and TSL are aware that inappropriate LOs can 
leave children demotivated. Overall, however, the schools reported positively in the second survey 
indicating that the intervention was effective in motivating the students (13 out of the 16 schools). 


Some staff also felt the sessions could be dull and repetitive, and suggested exploiting the potential of 
computer learning by using more maths games, brain warm-ups, league competitions, and avatars. The 
research also indicated that giving the children greater autonomy would improve their engagement. Both 
children and staff suggested that the children could have a more active role in choosing the LOs, and 
the children said they wanted to discuss the choice of methods with their tutors. They liked the active 
rewards, such as playing games and colouring in, and suggested choosing their own pictures. However, 
it is notable that even the children who made negative comments about the intervention mostly said that 
it helped their learning. 


Tutors 
Tutor-student relationships 


One of the greatest influences on the children’s dis/engagement was their relationship with their tutor. 
While some children enjoyed the novelty of talking to a person online, others expressed anxiety about 
having a relationship with someone they could not see. Many children said they wanted to be able to 
see their tutor, in a video or photograph. Several staff felt their children were more suited to classroom 
teaching with a teacher ‘there showing them... (and) communication around them and not directed 
straight at them’. One teacher said her children who had low self-esteem and negative attitudes to 
learning fared better with a classroom teacher who knew them. However, in some instances these 
anxieties were worked through, with staff supporting a child to re-engage. 


Learning experience 


The relationship between self-esteem and a positive learning experience was central, and played out 
through the children—tutor relationships. Some children clearly enjoyed the attention, support and 
encouragement of their ‘nice, kind’ tutors, who made them feel positive about themselves and their 
abilities. But others had negative relationships with their tutors and complained they interrupted, did not 
listen and pushed the child too hard when they did not understand. At worst some children complained 


that the tutors were rude and aggressive, which some staff affirmed. 
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The response when a child got something wrong was crucial as this fed into the children’s self-esteem. 
In positive scenarios the children felt supported by the tutor and encouraged to persist. Compared to 
the classroom, one child said 


they’d let you work it out again to see where you'd got it wrong. And | really enjoyed that because 
| didn’t have people shouting at me like ‘you've got it wrong, you’ve got it wrong’. 


However, several children complained that the tutor responded impatiently when they got stuck on a 
maths problem, and at worst found the tutor’s response intimidating 


| said | don't know, he was like ‘you need to think. What Is it? What is it?’... | just got a bit 
frightened. 


Just as praise affected positive engagement, negative feedback had the reverse effect. The complaints 
came largely from the children, although one teacher thought the tutors were overly strict and not 
sensitive enough. 


Difficulties 


Some children struggled with the tutors’ Indian accents, which was exacerbated when the children 
themselves had strong regional accents or speech impediments. When using chat box the tutors 
sometimes used slang language or misspellings. The slang language was often done intentionally in 
order to engage the children, but undermined the children’s comprehension and confidence in their 
tutors. For most children it took a few sessions to feel comfortable and confident with their tutor, so 
retaining the same tutor was important. However, many children had tutor changes, which could be 
disruptive and reduce engagement, although this could be an advantage when a child didn’t like their 
original tutor. 


TSL reported that education in India is more authoritarian than in the UK, and they invest in tutor 
professional development as a priority. They provide cultural and communication training, focusing on 
how to teach a child through encouragement and aim to recruit open-minded and flexible tutors. TSL 
also recognises that tutor retention is important for the continuity of children’s learning, and in order to 
impact this and improve tutor training they set up their own tutor centre in Sri Lanka (TSL Global) in 
September 2015, employing many full-time tutors, and increasing and improving teacher training. 


Formative findings 
The purpose of the formative findings is to inform the development of the intervention. 
Excellent features of successful delivery 


The process evaluation found a high incidence of perception of positive outcomes reported across the 
schools, from both staff and children. This included improvement in the children’s comprehension, 
mathematical capacities, verbal fluency, engagement, and confidence in maths. A key finding from the 
case studies was improved confidence, which grew alongside improved abilities and increased 
enjoyment of maths. This affirms Affordable Online Maths Tuition’s holistic approach to learning, with 
an ethos that emphasises the importance of enjoyment and self-confidence for engagement and 
achievement. The one to one element of the tuition was extremely beneficial for most of the children. 
This approach allowed for precision teaching and individualised learning. In the best cases it enabled 
the children to ‘have a go’ in a safe environment where they did not fear being thought ‘stupid’. This 
could increase the children’s self-confidence, and feed back into the classroom. Staff emphasised that 
many of the children would not otherwise have access to one to one tutoring. However, this excellent 
feature of the intervention fell down when the child did not get on with their tutor. 
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The intervention’s infrastructure, content and processes were good. Staff found the Affordable Online 
Maths Tuition interface easy to use, and reported clear explanations and modelling of methods, excellent 
visual slides, and appropriate content linked to the curriculum. The process of delivery was also an 
excellent feature of the intervention as the constant dialogue with the tutors improved the children’s 
verbal fluency, was a route to working out and understanding the maths, with a particular usefulness in 
relation to written word problems. TSL also gave excellent support, including technical and on-site 
support. 


An excellent feature of TSL as a start-up company is their ongoing commitment to identifying and 
addressing problems as they grow. To this end they invest in research and development, including the 
professional development of their tutors. This approach is affirmed in their practices as well as iterated 
in the interviews with themselves and Nesta, who work closely with them. Several of the improvements 
suggested by the case study analysis have already been addressed. During the trial year TSL has 
improved communication with schools, and instituted a mandatory orientation for both schools before 
the intervention starts, aimed at helping the staff and children understand how it works and how to best 
benefit. TSL has also improved process and curriculum design and employed a full-time operations 
team for visiting schools as well as employing more maths teachers and dedicated account managers 
for schools. 


Barriers and recommended modifications to successful delivery 


The technical problems were a barrier to the successful delivery of the online maths programme, 
although they improved once TSL re-installed the internet platform in the first weeks of the intervention. 
It is recommended that schools’ technological suitability for the intervention be reviewed before providing 
the intervention, and that TSL continue improving simplicity of processes and equipment quality, and 
continue offering high quality technical support and listening openly to schools. 


One of the side-effects of online tuition was that because the tutor was unable to see when a child was 
calculating a problem, they frequently interrupted the children’s concentration. A device indicating to the 
tutors that the children were working on a calculation would help, such as is available for other 
programmes, such as Skype or text messaging. Some children were anxious about relationships online 
with ‘strangers’, while others found their tutors authoritarian and insufficiently sympathetic. This 
impacted negatively on the children’s engagement. A change of tutor could also be destabilising and 
lead to disengagement. The tutors’ accent and speed of speaking could be a barrier, as could the use 
of slang/text language when writing. As discussed, TSL are addressing many of these issues through 
improved tutor professional development and setting up a dedicated tutor centre in Sri Lanka. Their 
teacher training programme has been developed with the Institute of Education in the UK and they have 
instituted a more robust English language test for tutors. TSL are currently planning research to better 
understand how the interaction between tutor and student, and also plan to personalise professional 
development for tutors. The introductory video that TSL have introduced for all pupils includes some 
discussion of what to expect from their online tutors, although this could include photographs or films of 
the tutors, and contextual information about them and their working environment. 


It is also recommended that TSL provide ongoing cultural professional development in a sympathetic 
teaching style, raising awareness of the children’s anxieties, using feedback from children that is elicited 
with the assurance of confidentiality to facilitate the children to speak openly. 


The staffs’ faith in the tutors’ abilities to set the correct LOs does not align with TSL’s emphasis on 
teachers setting the LO. It is helpful that TSL now have a formal orientation with staff in which they give 
guidance on setting appropriate LOs, including support for them to direct the learning objectives 
themselves. Ongoing evaluation of the intervention in schools should include assessing the 
appropriateness of the LOs chosen. However, given the staffs reliance on tutors setting the LOs, it is 
recommended that the tutors also be better trained to choose appropriate LOs and liaise with teachers. 
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A barrier to success was children becoming bored and disengaged, with a key factor being repetition of 
the same lesson for a particular LO. It is recommended that TSL offer a greater variety of lessons for 
the same LOs to enable embedding of a topic without children becoming bored and disengaged. Staff 
suggested having the LOs already covered on each child’s main page to help staff and tutors avoid too 
much repetition of the same lessons. Tutors could have further training in individualising lessons when 
LOs are repeated. 


Additionally, to better engage the children it is recommended that the potential of online learning be 
exploited, such as using more fun maths-games. Allowing the children greater autonomy and active 
ownership would also benefit their engagement, which could be achieved by allowing them input on 
selecting LOs, acting on their session feedback, and giving them more choices during the sessions. All 
the staff reported variation in outcomes for different children. So careful selection of children is 
recommended, by staff who know them well, and staff support for disengaged children before taking 
them off the intervention. 


The degree of staff investment in the intervention has an impact on how well it is implemented. Barriers 
to staff investment included time pressures, and staff turnover when new teachers were inadequately 
committed to the intervention. Occasionally poor communication with TSL impacted negatively. TSL 
have been doing research on how communication works in schools, and report that buy-in of schools’ 
senior leadership team supports the smooth running of the online service. TSL have extended the staff 
orientation to include new staff. TSL are planning further research to find out more about how schools 
work with TSL. 


We recommend that schools think carefully about the number and choice of staff running the 
intervention. Although Year 6 teachers were well positioned to make links between classroom and tutor 
teaching, the demands of their heavy workload meant that the intervention worked better when they 
were supported by a member of staff who had involvement in the children’s maths. The possibility of 
senior managers running all aspects of the intervention could be considered by schools. It is 
recommended that TSL improve sensitivity regarding the frequency, timing and recipients of reminders 
and emails. They should continue to listen to staff and take their comments on board. 


Future plans and how the programme is likely to function if taken to scale 


TSL reported that 12 of 32 intervention schools had bought the programme for the following year. During 
the process evaluation reasons given for continuing with the intervention were that it worked well and 
had a positive impact on the children’s learning, confidence, and self-esteem. Reasons given for not 
continuing included the cost, and preference for face-to-face interventions that were regarded as 
providing more individualised and effective teaching, as well as enabling easier and more 
comprehensive feedback to the classroom teacher. Typical of a start-up, Affordable Online Maths Tuition 
has been scaling up since it was launched in September 2013. When the trial began a year later in 
September 2014 the number of schools participating rose from 30 to 112. In September 2015 it rose 
again to 250. Scaling up is therefore a practical reality for TSL. Nesta reports that TSL are addressing 
the challenges of scalability well. 


TSL say one of the challenges of rolling the intervention out to more pupils and schools is continuing to 
recruit quality staff, including recruitment and training of good tutors at scale. This is met by TSL’s 
strategy of recruiting maths graduates from India, and more recently Sri Lanka. In these countries there 
is a good supply because teaching maths is a desirable, well-paid, and high-status job. 


Nesta notes that in general TSL continues to take on quality staff. TSL have increased staffing for sales, 
marketing, and support, which has helped address the logistical challenge of managing, monitoring, and 
tracking more pupils. 
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School staff identified the challenges if they were to roll the intervention out to more pupils: demands on 
staff time; access to computers and rooms: managing the sound in the intervention room if more pupils 
are doing it; timetabling; and keeping a tight rein on the LOs, especially for a shorter number of sessions. 


The staff felt that the intervention could be delivered effectively to different ability levels and year groups. 
Some staff felt the intervention would be more effective if introduced in earlier years in order to embed 
foundational learning at a crucial stage of learning. Staff also suggested the intervention span academic 
years, for example doing two terms in Year 5 and a final one in Year 6. 


Staff also highlighted technology problems as a potential barrier in scaling up, including the supply of 
broadband capacity for more pupils, and the difficulty of logging on more children. Nesta notes that 
technology in schools is variable, with different problems for different schools, such as lack of technical 
expertise, poor infrastructure, and old buildings with thick walls interfering with connections to the 
internet. They say that connectivity is challenging in schools, with variable broadband speed and 
bottlenecks with internal networks impacting on individual connections. Investment in broadband speed 
is variable, especially in primary schools that are often not large enough to get sufficient funding. 
However, Nesta reports that TSL ensure their technology can scale by keeping the complexity of 
equipment low. 


Cost 


The programme cost £378 per pupil for equipment and 27 weeks of tuition. The average cost per pupil 
is £378 per year over three years. 


Year1 | Year 2 Year 3 
Service fees per pupil £378 £378 £378 


Cost per pupil per year over three years £378 


Time costs 


There was some time required at the beginning of the year to set up the intervention. On average, 
teachers spent 10 minutes per pupil to create their academic profile and account. It took 90 minutes on 
average to set up and test the local computers for a class. Given that on average there are 10 pupils 
per class, this works out at 9 minutes per pupil. 


It took teachers about 25 minutes to pick the lesson each week for the group. 


Someone needs to oversee the sessions, which last 45 minutes in total. Normally, this is a TA and it is 
mostly light-touch observation, allowing them to do marking or other tasks alongside. 
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Conclusion 


Key conclusions 


. The impact evaluation found no evidence that the intervention had an impact on Key Stage 2 
maths, compared with ‘business as usual’ teaching and support in Year 6. 


. Teachers were largely positive about the online tuition, and reported that it appeared to improve 
pupils’ comprehension, verbal fluency, and confidence in maths. 


. Schools should consider whether their computer network can support the implementation of an 
online programme. Teachers were positive about the technical support and user experience of 
the programme, but some experienced technical challenges in the implementation. 

. As the online tuition is a ‘talking’ intervention, it appeared to work better when the pupils were 
spaced out in larger rooms so that the noise from other sessions was less distracting. 

. Future research could examine whether the programme has an impact on pupils’ comprehension, 
mathematical capacities, verbal fluency, and confidence in maths, as this was an outcome 
reported by teachers. 


We undertook a large pragmatic randomised controlled trial of Affordable Maths Tuition with an 
embedded process evaluation. The trial was robust and followed CONSORT standards and the process 
evaluation examined the perspectives of teachers, pupils and staff of the delivery organization. 


Impact evaluation 


In the impact evaluation we found no evidence of an effect of the intervention. The effect size was close 
to zero and this did not change when the analysis used the pre-specified approach adjusting for key 
baseline variables or using an unadjusted analysis. 


Process evaluation 


The process evaluation found that Affordable Maths Tuition provided an excellent holistic learning 
experience via one to one tuition that individualises learning and has the potential to accelerate it. 
Schools were largely positive about the online tuition, confident that the tuition was beneficial for their 
pupils’ outcomes in terms of improved comprehension, verbal fluency, and confidence in maths. 


TSL is committed to learning from mistakes and has already instituted many improvements that this 
report recommends. The intervention is well supported by TSL, including technical and on-site support. 
It is recommended that TSL continue to invest in tutor professional development, including cultural 
training to develop a sympathetic teaching style. It is also recommended that TSL continue developing 
the intervention so it engages the children, including providing more variety of lessons and LOs, 
exploiting the potential of online learning, such as developing more maths games, and facilitating more 
active participation for the children in the intervention. 


Summary 


There is no evidence from this trial that the intervention Affordable Maths Tuition is an effective method 
of improving KS2 maths scores compared with ‘business as usual’ and, therefore, we cannot 
recommend that schools purchase the intervention with the intention of improving this outcome. There 
may be several reasons for the lack of effect observed in the trial. Some schools in the intervention 
group (n = 6) implemented one to one tuition using face-to-face tutors. This may have diluted any 
intervention effects; however, 80% of the control schools did not implement such a scheme. 
Consequently, we might still have expected to see some evidence of an effect even if it were somewhat 
diluted. In the process evaluation there were some issues that may have reduced the impact of the 
intervention to achieve any potential for effectiveness. There was the lack of face-to-face contact with 
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the pupils and some complained of being interrupted when working, as the tutor could not see that they 
were still working on a problem. Furthermore, there were some reports of difficulty in understanding 
accents and differences in expectations which may also have contributed to the null finding. There were 
some short-lived technical issues that also may have played a role in limiting impact. 


Impact evaluation: Strengths 


e Arobust RCT design. 

e Robust trial conduct and reporting. 

e Reasonable compliance to the intervention. 

e An educationally significant outcomes measure. 


Process evaluation: Strengths 


e Arrange of data collection methods used. 
e Data collected at different time points. 
e Robust sampling of case study schools. 


e Data collected from a range stakeholders including students, teachers, teaching assistants, and 
the developers. 


Impact evaluation: Limitations 


e Powered only to detect an effect size of 0.33. 


e Evidence of non-compliance in the control group, with 20% of schools engaging in one to one 
tutoring and a further 10% in online learning. 


Process evaluation: Limitations 


e Findings from the case study interviews and focus groups cannot be generalised. 
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Appendix A: School Attendance Proforma 


ee 
_ A tion ww 
UNIVERSITY Of York WDucham Nesta.. TmIRO SPACE 


Education Endowment Foundation Online Maths Tutoring Project 
Information meeting for headteachers 
1.30pm to 3.00pm on [Day] [Date] March 2014 


[Location — full address] 


| confirm that a school representative will be attending the information event. 


Representative’s Name: 


Email: 
School: © 0000000000 
Head Teacher: OE 


Please return this proforma (preferably by fax — 01904 321387) to Natasha 
Mitchell by [Day] [Date]*" March 2014 at the latest. Your prompt reply will be 


much appreciated. 


York Trials Unit, Lower Ground Floor, ARRC Building, Department of Health Sciences, University of York, 
Heslington, York, YO10 5DD. Tel: 01904 321655 Email: natasha.mitchell@york.ac.uk 


Information Event School Proforma v1.1 29Jan14 
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Appendix B: School Expression of Interest Document 
A 


AB 
UNIVERSITY of York Ww Durham Nesta.. Taino sence 


An Evaluation of an Online Maths Tutoring Intervention: School Expression of Interest 


We are currently seeking expressions of interest from schools to participate in an evaluation of an online 
maths tutoring intervention. In this evaluation schools will receive the intervention provided by Third 
Space Learning free of charge. 

The Education Endowment Foundation has asked researchers at the University of York and Durham 
University to evaluate the online maths tutoring intervention developed by Third Space Learning and 
supported by Nesta in 2014 and 2015. The aim of the evaluation is to find out if the intervention helps 
to improve pupils’ maths skills during year 6, especially the maths skills of those pupils who are 
struggling. 


In the online maths tutoring intervention, online tutors trained in the National Curriculum, and based in 
India, provide one to one support for pupils during sessions lasting one hour. Class teachers identify 
areas of development for each pupil and select modules which help to address these needs. Online 
sessions are available throughout the school day. Pupil welfare is maintained throughout the Third 
Space Learning programme, with all online maths tutors vetted by Third Space Learning and holding a 
police clearance certificate (the Indian equivalent of the UK DBS check). All sessions are recorded and 
teachers have access to all recordings for their pupils. No tutors have access to any personal pupil data. 


There will be two groups of primary schools in the evaluation. Allocation to these groups will be decided 
by random selection (like in a lottery). Both groups of schools will be asked to identify 10 pupils that will 
be attending year 6 in the Autumn Term of 2014 and anticipated to achieve KS2 level 3 or a borderline 
KS2 level 4. 


Group A schools will implement the online maths tutoring intervention in Autumn Term of 2014 with 
those 10 pupils identified as meeting the criteria for the study. Group B schools will not receive the 
intervention in 2014, but will be offered the online maths tutoring intervention free of charge in the 
Autumn Term of 2015. The researchers at the University of York and Durham University will then 
compare the KS2 results of pupils from schools in both groups at the end of year 6 2014/5 to estimate 
the effect the intervention has had on pupils’ maths skills. 


What commitment would this require from schools? 
e Enthusiasm for the project and for your own professional learning 
e Provision of baseline data about pupils in year 5 (in May term 2014) 
e Willingness to allow random allocation to the ‘online tutoring’ intervention in 2014 or 2015 
e Willingness to identify 10 year 5 (in May 2014) pupils plus 3 reserve pupils 
e Attendance at the project information event 
e Willingness to implement the intervention only to those identified 
e Willingness to follow the guidance provided by the researchers 
e Provision of a designated space for online tuition sessions for pupils 
e Reliable internet connection 


When will this project take place? 

We hope to hold information events in March 2014 and to randomise participating schools in early June 
2014. Primary schools, who are allocated to implement the intervention in 2014, will begin the online 
tutoring after the summer holidays. 
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Please come to the information meeting to find out more 
On [Dates of Events 2014] the Evaluation team, Third Space Learnings and Nesta will jointly hold an 
information meeting for schools to find out more about the intervention and its evaluation. We very much 


hope to see you at this event — [TIME & LOCATIONS]. 


For further information about this study or to book a place, please contact: Dr Natasha Mitchell at the 
University of York. Email: natasha.mitchell@york.ac.uk; Tel: 01904 321655. 


Principal Investigators: 
Professor David Torgerson, York Trials Unit, Department of Health Sciences, University of York, Heslington, York, 
YO10 5DD. T: 01904 321340 E: david.torgerson@york.ac.uk 


Professor Carole Torgerson, School of Education, Durham University, Leazes Road, Durham, DH1 1TA. T: 0191 
334 8382 E: carole.torgerson@durham.ac.uk 
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Appendix C: Primary School Agreement to Participate 


AB A 
UNIVERSITY f York W Durham Nesta.. 


University 


Evaluation of Third Space Learning Online Maths Tutoring Intervention 


Primary School Agreement to Participate 


(please tick) 


| confirm that | have read and understood the information sheet for the above evaluation and have had 
the opportunity to ask questions. 


| understand that all children’s results will be kept confidential and that no material which could identify 
individual children or the school will be used in any reports of this evaluation. 


| agree to send an information letter out to all parents/carers of children in Year 5 (in May 2014) and collect 
in any returned opt out forms. 


l agree to provide baseline data (including UPN, DoB) about pupils in Year 5 (in May 2014) to the evaluation 
team and EEF (excluding any pupils for whom opt out forms have been returned). 


| understand that named baseline data will be matched with the National Pupil Database/Pupil Matching 
Reference and shared between the evaluation team and EEF. 


| agree to random allocation to implement the ‘Online Maths Tutoring’ intervention in 2014 or 2015. 
| agree to identify 10 pupils, plus 3 reserve pupils, who may be allocated to receive the intervention. 
| understand the intervention should only be given to pupils which have been identified. 

| understand we should provide a designated space for tuition sessions. 


| consent to the school taking part in the above study. 


Name of Headteacher: 
Name of School: 

School Tel no: 

Headteacher Email address: 


Name of School Contact (if not Headteacher): 
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School Contact email address: 


Signature of Headteacher: 


Thank you for agreeing to take part in this research. Please return this consent form at the information 
meeting or afterwards by post to: 


Dr Natasha Mitchell, York Trials Unit, Lower Ground Floor, ARRC Building, Department of Health 
Sciences, University of York, Heslington, York, YO10 5DD. 
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Appendix D: Parent Information Letter & Opt Out Form 


A 


AB 
UNIVERSITY of York W Durham Nesta.. THIRD SPACE 


Dear Parent / Carer 
Your child’s school is taking part in the Online Maths Tuition evaluation. Durham University and the 
University of York have been asked by the Education Endowment Foundation (an organisation funding 


research into education) to independently evaluate the Online Maths Tuition programme provided by 
Third Space Learning. 


The Online Maths Tuition programme has been developed by Third Space Learning and supported by 
Nesta (a charity which helps organisations develop new ideas). It is designed to improve children’s 
maths skills, especially those who struggle with maths. Good maths skills are important for all children. 


To find out how well the Online Maths Tuition programme works some schools will use the Online Maths 
Tuition programme this year and some schools will not. This is decided randomly by a computer 
(however all schools will continue to teach children maths skills). Researchers will then compare results 
from schools that have used the programme with schools that have not. In order to do this we would like 
to collect information about your child from your child’s primary school. 


Your child’s school will provide information including your child’s name, date of birth, gender, unique 
pupil number, details on your child’s current National Curriculum maths level and free school meal 
status. 


Your child’s information will be treated with the strictest confidence. Named data will be matched with 
the National Pupil Database and shared between the evaluation team and the Education Endowment 
Foundation. We will not use your child’s name or the name of the school in any report arising from the 
research. Your child’s information will be kept confidential at all times. 


If you are happy for your child’s information to be used you do not need to do anything. Thank 
you for your help with this project. 


If you would rather your child’s school did not share your child’s information for this project please 
complete the enclosed opt out form and return it to your child’s school by [INSERT DATE]. 


If you would like further information about the Online Maths Tuition evaluation please contact Natasha Mitchell 
the Evaluation Coordinator: natasha.mitchell@york.ac.uk; 01904 321655. 


Yours faithfully 


Professor David Torgerson (University of York) 
Professor Carole Torgerson (Durham University) 
Nesta 

Third Space Learning 

Education Endowment Foundation 
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ee 
ae A | we 
UNIVERSITY OfYork en Nesta.. Tmino sence 


Online Maths Tuition Evaluation: Opt Out Form 


If you DO NOT want your child’s data to be shared for use in the Online Maths Tuition evaluation, 
please return this form to your child’s school asap. 


|| | DO NOT want my child’s data to be shared for use in the Online Maths Tuition evaluation 
Parent/Carer Sign ature......c.ccccccccsssssssscsecscsesseecsesesscscsecassecsesceeseesesacacsecasseseeeceeeees DatC aces sete tate ttectastssenvecetencess 


CHINAS NAIM ies secccscetesteceied ccdesoeetaxcedtcebvctes isccsseacusa cevbusteysenccossncedietleaduiacsducb eset ects cde vlueececcndeselsonsieviieeabu stesdascesPeadeens 


GHildi'S!SCHOO lccscccceissciecscckechsd5icaced ccd acesdesteysnnceesneeeackess sasdevenccbsstaceeceevedsvascesspacd gant duakaessba pescelansanedeusveysioenecscesschayvests 
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Appendix E: Process Evaluation: Top Tips for Schools 


Top Tips for Schools PO 


Project a member of the Evaluation Team visited a number of 
schools in November/December 2013. The purpose of these visits 
was to spot practical ideas for new schools using the online 
maths tool. We have collated these ideas and developed a Top 
Tips’ sheet. The suggestions below are all from schools who have 
been using the online maths tool. 


As part of the process evaluation for the Online Maths Tutoring ~e 


« Schedule the online tutoring during the normal school day and outside 
regular mathematics lessons. 


|. A teacher should be responsible for choosing the mathematics topic. It 
is recommended that topic selections are based on gaps in the pupil's 
knowledge and that the online tutoring provides additional support for 
the pupil. 


- It is important to spend time upfront setting up the booking system. 


« Book topics for each session well in advance but at least 24 hours prior 
to the session taking place so that the tutor has time to prepare. 


}« Completing the online tutoring in a computer room is highly recom- 
mended. If not available, then another room or space away from other 
pupils would be suitable. 


+ Brief the pupil on how the online tutoring will work prior to their first 
session. This does not have to be extensive but just tell them what to 
expect. They will become very comfortable with the set up very quickly. 


« Brief the pupils on a plan of action for when technical issues such as a 
poor intemet connection, loss of audio and faulty headsets occur. You 
will find that the pupils will easily troubleshoot issues themselves once 
they start using the invention regularly. 


|. It is recommended that the online tutoring session be supervised. 


OMT Jogtys v1.1 21Aug1s 
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Appendix F: Process Evaluation: Questionnaire 
Intervention Schools 


AFFORDABLE MATHS TUITION-Autumn Survey 


Your students are part of the Online Maths Tutoring Project funded by the Educational Endowment 
Foundation. By now you are up and running using the online tutoring provided by Third Space Learning 
(TSL). As you have been previously notified by the York evaluation team, we are conducting a process 
evaluation alongside the randomised controlled trial. The purpose of the evaluation is to collect data 
to understand more about how online tutoring is running in order to identify issues that might need 
addressing if more schools opted to use this tool in the future. We are interested in knowing what has 
worked well for you and your students in the first few weeks and what has not. This questionnaire 
has 12 questions and will take less than 10 minutes to complete. The questionnaire is organised into 
three key sections. 


I. Communication 
Il. Technical issues and support 
ll. Teaching and Learning 


By completing this survey, you will be agreeing to participate in this small study. Your responses will 
be treated confidentially and the data will be held securely. Thank you in advance for your co-operation 
and time. If you have any questions or technical issues concerning the survey please email Gillian at 
g.hampden-thompson@york.ac.uk. 
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Q1 SECTION |. Communication Did you personally attend one of the initial TSL regional recruitment 
events that took place in the Spring 2014? 


L) Yes 
L} No 


Q2 Were the session administrator and IT lead briefed prior to the trial starting? 


O Yes, by myself 
O Yes, by someone else 
OQ No 


Q3 How satisfied were you with the level of communication you received from TSL prior to the trial 
starting? 


O Satisfied 
O Neither satisfied nor dissatisfied 
O. Dissatisfied 


Please indicate specific issues/comments below: 
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SECTION II. Technical issues and support 


Q4 Please indicate your level of agreement with the following four statements. 


Strongly Dleaeree Neither Agree 


Disagree nor Disagree peice Strongly Agree 


It was easy to 
setup the TSL | O O O O O 
account. 


It was difficult 
to set up the 
student 
profiles. 


All other 
aspects of the 
ITsetup were | O O O O O 
straight 
forward. 
The ongoing 
technical 
support from | O O O O O 
TSL has been 
good. 

Q5 What were the most common technical issues the students experienced with the online tutoring 


in the first few weeks? 


OQ Faulty headsets 

OQ Poor internet connection 

QO. Difficulty understanding the tutor 
O Other (please specify below) 


Q6 Do you feel that your IT support lead in your schools has the sufficient skills to provide ongoing 
support for the online maths tutoring? 


O Yes 

OQ To acertain extent 

OQ No, they would benefit from additional training 
Please indicate specific issues below: 
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SECTION III. Teaching and Learning 
Q7 Were students given any sort of training prior to starting the online maths tutoring? 


O Yes 
O No 


Q8 Did you and your teaching colleagues discuss or plan how the online maths tutoring might be used 
in conjunction with existing teaching approaches? 


O Yes 
O No 


Q9 How often do you feel that the TSL academic learning objectives match up with the students’ own 
individual learning objectives as set by you or your school? 


OQ For all students 
OQ For most students 
OQ Fora few students 
OQ Forno students 


Q10 Academic feedback is given after each session. How useful have you found this feedback? 


O Very Useful 

Q Somewhat useful 

O Not useful 

OQ Did not know there was feedback available 
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Q11 Please indicate your level of agreement with the following four statements. 


The students 
said they have 
enjoyed the 
online maths 
tutoring. 


The students 
appear 
disengaged 
during their 
tutoring 
sessions. 


The students 
display more 
confidence in 
maths asa 
result of the 
online 
tutoring. 


| am so far not 
convinced 
that the 
online 
tutoring is an 
effective 
teaching 
method. 


strongly 
disagree 


Disagree 


Neither Agree 
nor Disagree 


Agree Strongly Agree 


Q12 We would welcome any additional comments you may have concerning the online maths tutoring 


trial. 


This is the end of the questionnaire. Thank you for your co-operation. 
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Appendix G: Process Evaluation: Questionnaire 
Intervention Schools 


AFFORDABLE MATHS TUITION-Summer Survey 


Your students are part of the Online Maths Tutoring Project funded by the Educational Endowment 
Foundation. As you will be aware, the year-long trial is coming to an end. As you have been previously 
notified by the York evaluation team, we are conducting a process evaluation alongside the 
randomised controlled trial. As we have previously indicated, the purpose of the evaluation is to collect 
data to understand more about how online tutoring is running in order to identify any issues that might 
need addressing if more schools opted to use this tool in the future. Therefore, we would like to ask 
some final questions of your experience with the online maths tutoring 


This questionnaire has 12 questions and will take less than 10 minutes to complete. The questionnaire 
is organised into three key sections. 


lI Pupil achievement 
ll. Pupil engagement 
Hl. Future plans 


By completing this survey, you will be agreeing to participate in this small study. Your responses will 
be treated confidentially and the data will be held securely. No school or teacher will be identified in 
the report or other disseminated research outputs. Thank you in advance for your co-operation and 
time. If you have any questions or technical issues concerning the survey please email Gillian at 


g.hampden-thompson@york.ac.uk. 


SECTION I: Pupil achievement 
Q1 Did you know about/had you noticed any impact of the intervention on pupils’ achievement? 


L) Yes 
L} No 


Q2 How have you assessed the impact of the intervention on pupils’ achievement? (please tick any 
that apply) 


O talking to the pupils 
O reading pupils feedback online 

O listening to the audio-report of online sessions 
OQ observing its impact on pupils maths practices 
O 


Other (please specify below) 
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Q3 Please indicate your level of agreement with the following statement. 


Strongly Disagree Neither Agree Strongly Don’t know 


Disagree Agree nor Agree 
Disagree 


The intervention 
was effective in 
improving the 
children’s skills in 
maths 


The intervention was 
not effective in 
improving the 

children’s 
achievement in 
maths 


Q4 Was there much variation in the effectiveness of the intervention 


OQ Between pupils 

O Less effective over time 

OQ More effective over time 

If any of the above, please specify 


Q5 Was there anything else supplementary done with these or other particular groups of children, 
either through the school or the parents? 


L) Yes 

CO) No 

QC) Don’t know 

If yes please indicate what was done, by whom and with which groups: 
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SECTION II: Pupil Engagement 


Q6 Please indicate your level of agreement with the following two statements. 


Neither 
Agree nor 


Strongly 


Disagree Agree Strongly 


Disagree Agree 


Disagree 


The intervention 

was effective in O Oo O Oo O 

motivating the 
students. 


The children were 
not enthusiastic 


; O O O O O 
about doing the 
online intervention 
The intervention 
fective i 
was effective in O O O O O 


raising the children’s 
self-confidence 


IDYoyaya el dalen 


Q7 Did the pupil’s behaviour change in maths classes (or elsewhere) as a result of doing the maths 


intervention? 


L) Yes 
CL) No 
Please indicate behavioural changes below: 


Q8 Did the pupils generally manage to attend the whole hour 
L) Yes 

L) No 

If not, please indicate why this was the case: 


Q9 Were there any disadvantages to the intervention being one-to-one? 


L) Yes 

C1 No 

C1 Don’t know/Not sure 

Please indicate specific issues below: 
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Q10 Were there any advantages to the one-to-one intervention being delivered online via a computer? 


L) Yes 

O) No 

LJ Don’t know/ 

Please indicate specific issues below: 


SECTION III: Future Plans 


Q11 Please indicate your level of agreement with the following three statements. 


Strongly Disagree Neither Agree Strongly Don’t know 


Disagree Agree nor Agree 
Disagree 


It would not be a 
good idea to carry 
on with the 
intervention in the 
future 


The intervention 
would work better if 

it could be done at OQ Oo Oo Oo O O 
various times of day 


The intervention 
would work better if 
delivered for 
different lengths of 
time as and when 


pupils needed it 
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Q12. Please indicate your level of agreement with the following four statements 


Strongly Disagree Neither Agree Strongly Don’t know 


Disagree Agree nor Agree 
Disagree 


It would be a good 
idea to roll the 
intervention out to 
pupils with lower 
abilities 


It would be a good 
idea to roll the 
intervention out to 
pupils with mixed 
abilities 


| do not think it 
would be a good idea 
to roll the 
intervention out to Oo Oo O O Oo O 
higher achieving 
pupils 


| do not think it 
would be a good idea 
to roll the 
intervention out to OQ oe) Oo O Oo Oo 
pupils in different 
year groups 


Q13 what would the main challenge of rolling the intervention out in the future be: 


OQ Affordability 

OQ Organising the pupils time 
OQ Taking up staff time 

OQ Other 

If other, please specify below 
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Q14 Could any improvements be made? 


O Yes 
QO No 
OQ. Don't know/Not sure 


If yes, please specify below 


Q.15 We would welcome any additional comments you may have concerning the online maths tutoring 


trial. 


This is the end of the questionnaire. Thank you for your co-operation. 
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Appendix H: Process Evaluation: Questionnaire Control 
Schools 


Your students are part of the Online Maths Tutoring Project funded by the Educational Endowment 
Foundation. As you are aware, your school was selected as a ‘control’ school so the 10 pupils who were 
selected did not receive the online tutoring intervention. As part of the randomised control trial, the 
York Trials Unit who are conducting the evaluation need to check with every control school that you 
did not put in place a similar invention for the 10 pupils that are part of the trial. Can | ask you to 
answer the two questions below and send me your answers by simply replying to this email? Please 
email me if you have any questions. 


1. For the 10 pupils that are part of the trial, can you state whether any additional mathematics 
tutoring was given to the pupils during the last academic year? 
(Delete as appropriate) 
Yes 
No 
Unsure 


2. If you answered ‘unsure’ above, please indicate why you are unsure. For example, is it possible 


that their parents provide additional maths tutoring outside of school hours? Please include 
any information you have below even if you think it might not be directly related. 
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Appendix I: Process Evaluation: Teacher Interview 
Schedule 


Interview time: approx. 45 minutes. 

Type: semi-structured. The questions are open with prompts in italics. 
A. Introduction 

1. Which staff were involved in setting up and running the session? 

2. What is your general feeling about the intervention? 


e What have other staff and the pupils said? Parents? 
B. Logistics 


3. We know children on pupil premium who were insecure/level 4 were selected, but were there any 
other factors involved in selecting the children? 


e Any issues with selection? (Eg with pupil premium group targeted) 
e Were the selected children always the ones doing the online sessions? 
e Did the pupils receive any additional extra tutoring inside or outside of school (to the best of 
your knowledge)? 
4. Where and when did the sessions take place? 


e After school, lunchtime, during lessons? 
e Ownroom or with others? How many children at a time? 
e How did this work? Any issues? 

5. What did you have to do to set up and run the intervention? 


e £g booking sessions, monitoring progress; listening to audio reports (prompt if gaps in 
AFFORDABLE MATHS TUITIONforms) 
e How were the learning objectives set? Who chose them? 
e How much time did setting up and running it take? 
e How did you feel about being involved in the intervention? Any issues? 
6. Were there any issues with the technology? For you and the pupils? 


e Eg. Logging in, audio, whiteboard, accessing audio-report, tutor feedback 
7. Do you know how the pupils found using the intervention? 


e Topic language or different ways of doing maths calculations? 
e The accent 
e Filling in the self-assessment 
e Did they manage the whole hour each time? If pupils late/left early, why? 
8. Did you and the pupils have sufficient training and support before and during the intervention? 


D. Effectiveness of Intervention 
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13. How effective do you think the intervention was? Have you noticed any changes in the pupils? 


e Maths capacities and attainment 

e Interest and engagement 

e = Self-confidence and behaviour 
14. Was there variation? 


e Between pupils 
e Change over time 
15. How did you know about the impact of the intervention on pupils? How did you get feedback? 


16. What are your thoughts on the one-to-one aspect of the online tutoring? Compared to the 
classroom experience? Eg. no eye contact? not having to perform in front of/against others? 


17. Was there anything else supplementary done with these or any other particular groups of children? 
eg to prepare for SATS? Eg from parents? 


E. Scaleability 
18. What are your thoughts about taking the intervention forwards? 


e Good idea or not a good idea? 
e Same or different children? More children? Different years? Different abilities? 
19. How might you use it? Eg. as in the trial or in more flexible ways 


e Same, different or varying length of time? 
e Eg cramming for SATS 
e Different times of day? 
e Different for different pupils? 
20. What would be the challenges of rolling out intervention to more pupils? 


e = £g practical: management; technical; pupil/staff time; fitting round lessons 
e Eg funding 
F. Final questions 


21. Has the school carried out its own assessment of the intervention? 
22. Could any improvements be made (n.b. question left purposively open)? 


23. Your overall feeling about the intervention? 
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Appendix J: Process Evaluation: Children Focus Group 
Schedule 


Focus group time: 

The focus group schedule consists of 6 open questions, with prompts bulleted below in italics. 
A. Logistics 

1. Where and when did you do it? 


e Were you taken out of lessons? After school? Lunchtime? Did you mind this? Why/why not? 
e How was this? Would there have been a better time or place to do it? 
e Were there any distractions eg background noise? Was this a problem? 
e How did you know when to do it? Were you reminded? How did that work? 
e Were you able to get there on time and do the whole hour? Prompt arrive late/leave early 
depending on data) 
2. What did you have to do? Was it easy to access the sessions? 


e £g logging on, when doing the sessions? Talk me through how it all worked? How was the 
audio? The whiteboard? Any issues? If so, what did you do? Was there anything about how you 
interacted with the technology that was helpful or annoying? (eg able to draw, enter or day 
number) 

e How easy was it to use the computer and connect and communicate online? 

e Was the audio a problem? 

e Was the whiteboard a problem? 

B. Evaluation of Intervention 


3. What did you think of the online maths tutoring sessions? 


e What did you like/dislike about doing the online maths sessions? 

e Did you find the sessions interesting or fun or boring? Could you concentrate? 

e Did they help you? What was/ wasn’t helpful? 

e Do you feel any differently about doing maths? Do you feel more or less confident? 

e What do you think of it compared to your usual maths classes? Eg. one to one; using computer. 
4. Can you tell me about your tutors? 


e How did you find the tutors? 
e How did you communicate with the tutors? 
e Could you understand the tutor ok? eg accents; Explanations. 
e Was having different tutors a problem? 
5. Did you know how you were doing? 


e Did you get feedback on your learning? How? Was this helpful or not? 
e How did you do? 
C. Future use of the intervention 
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6. Would you want to do it again? 


e What would make it a better experience for you? 
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Appendix K: Process Evaluation: Intervention Provider 
Interview Schedule 


Themes to be covered in unstructured/semi-structure interview 


Set up and implementation 

Student engagement 

Communication 

Monitoring and responding to feedback 
Specific schools and specific issues 
Technology 


So. or Pow ee 


Going to scale 
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Appendix L: Process Evaluation: Teacher Information Sheet 


Finding out how to support and improve 
children’s maths skills 


The Online Maths Tuition Project: A research 
study 


UNIVERSITY Of York 97 Durham » naowmen 


ee 
Nesta... sas 


Information for Teachers 


OMT_Teacher_PIS v1.1 19May15 
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What is research? Why is this being done? 

Research is a way we try to find out answers to questions. 
Maths is an important skill that everyone needs to have. 
Maths can be hard and some children need help to improve 
their maths skills . We are trying to find out if using an 
online maths tutor can help children with their maths. 


How does the research work? 

Your school is taking part in this project. During Year 6 a 
number of pupils have had the opportunity to use the online 
maths tutor to improve maths skills. We would like to talk to 
you about the use of the online tutor at your school. We are 
interested to hear your views including, the impact it has 
had, ease of use, problems encountered for example. To do 
this we would like to have a one to one discussion with you at 
your school. 


What will I have to do? 

If you decide to take part you will need to be willing to be 
interviewed by one of our researchers. We will record the 
interview and transcribe it afterwards. We anticipate the 
interview to last between 30— 60 minutes. 


Will anyone else be told about what I say? 
Everything you say will be kept confidential and 
anonymous. . 


Do I have to take part? 

You can decide if you want to take part. If you don't want to 
take part you do not have to give us a reason. If you decide 
to take part now you can always change your mind later on. 
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Did anyone check to see if this study is OK to do? 

Before any research study is allowed to happen it has to be 
checked by a group of people called a Research Ethics 
Committee. They make sure that anyone who takes part is kept 
safe and the research is fair. This study has been checked by 
2 committees, one at the University of York and one at 
Durham University. 


Who is running the study? 
The study is being run by the University of York and Durham 
University. 


What happens next? 
If you want to take part, please complete the consent form 
and return this to the researcher.. 


Who can I speak to about the study? 
You can speak to or email either: 


Debbie Humphry who will be carrying out the interviews 
t: 07831 811490;e: D.V.Humphry@sussex.ac.uk 


Natasha Mitchell who works for the University of York 
t: 01904 321655; e: natasha.mitchell@york.ac.uk 


If you are unhappy about the project you can contact 
Gillian Hampden-Thompson who is leading it. 
t: 01273 872601; e: G.Hampden-Thompson@sussex.ac.uk 


70 


Affordable Online Maths Tuition 


Appendix M: Process Evaluation: Teacher Consent Form 


Teacher Consent Form 


If you would like to take part in the Online Maths Tuition Project Teacher Interviews, please complete 
the following information and return this form to the researcher. 


Please initial each box 


1. | confirm that | have read and understood the information sheet for the above project and 
have had the opportunity to ask questions. 

2. | understand that the interview will be recorded & transcribed. [| 

3. | understand the recordings & transcriptions will only be accessed by researchers working on | | 
this project 

4. | understand anything shared during the interview will be treated with the strictest | | 
confidence and my name or school will not be used in any reports arising from the 
research. 

5. | understand that the information gathered will be used to write research articles and | | 
reports, but will not identify me by name. 

6. | understand that my participation is voluntary and that | am free to withdraw at any time [| 
without giving any reason. 

7. | CONSENT to taking part in the Online Maths Tuition Project Teacher Interviews [| 

Name Date Signature 

Name of Researcher Date Signature of Researcher 


UNIVERSITY Of York Durham i nsowmen Nesta.. THAD SPACE 


University oundation - 


Online Maths Tuition_Teacher Consent Form v1.1 19May15 
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Appendix N: Process Evaluation: Pupil-Parent Information Sheet 


What happens next? 

If you want to take part, please complete the consent form 
and take this back to the class teacher at school. Please 
tell your parent or carer if you don't want to take part. 


Who can I speak to about the study? 
You can speak to or email either: 


Debbie Humphry who will be holding the discussion groups 
t: 07831 811490 
e: D.V.Humphry@sussex.ac.uk 


Natasha Mitchell who works for the University of York 
t: 01904 321655 
e: natasha.mitchell@york.ac.uk 


If you are unhappy about the project you can contact 
Gillian Hampden-Thompson who is leading it. 

+: 01273 872601 

e: G.Hampden- Thompson@sussex.ac.uk 


Education Endowment Foundation 


Finding out how to support and improve 
children’s maths skills 


The Online Maths Tuition Project: A research 


study 


Information for Parents and Year 6 children 


OMT_Parent-Child_PIS vi.119Mayi5 
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What is research? Why is this being done? 

Research is a way we try to find out answers to questions. 
Maths is an important skill that everyone needs to have. 
Maths can be hard and some children need help to improve 
their maths skills . We are trying to find out if using an 
online maths tutor can help children with their maths. 


How does the research work? 

Your school is taking part in this project. During Year 6 you 
have had the chance to use the online maths tutor to 
improve maths skills. We would like to talk to some of the 
children who have used it to see what they think about it. 
We will ask about what you enjoyed, what you didn't enjoy 
and suggestions of how it could be made better for other 
children. To do this we would like to have a discussion 


group at your school. You will be with other children who 
have used the online maths tutor. 


How does helping effect the project? 

We want to talk to children, like you, who have used the 
online maths tutor. By taking part in the discussion group 
you will be helping us to understand the good and bad 
points. We can then use this information to change things 
for future children who use the online maths tutor. 


What will I have to do? 

If you and your family decide to take part you will need to 
come to a discussion group at your school. At the 
discussion group we will ask some questions about using the 
online maths tutor. We will record the discussion group so 
that we can type up all the things that were talked about. 
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The discussion group will last about 
3O0minutes. 


Will anyone else be told about what I 

say? 

Everything you say will be kept 

confidential. This means that we will 

not share what you say with other 

people. If we do share anything this 

will be anonymous, which means people won't know who said it. 


Do I have to take part? 

You can decide if you want to take part. If you don't want to 
take part it won't change things at school, and you do not 
have to give us areason. If you decide to take part now you 
can always change your mind later on. Just let your parent, 
carer or teacher know that you don't want to take part any- 
more. 


Did anyone check to see if this study is OK to do? 

Before any research study is allowed to happen it has to be 
checked by a group of people called a Research Ethics 
Committee. They make sure that anyone who takes part is 
kept safe and the research is fair. This study has been 
checked by 2 committees, one at the University of York and 
one at Durham University. 


Who is running the study? 
The study is being run by the University of York and Durham 
University. 
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Appendix O: Process Evaluation: Parent-Child Consent 
Form 


Parent-Child Consent Form 


This form needs to be completed by the child and their parent/guardian. 

If your child would like to take part in the Online Maths Tuition Project Discussion Group, please 
complete the following information and return this form to your child’s class teacher. 

To be completed by the parent/carer 


Parent/Guardian Name: 


Please initial each box 
1. I confirm that | have read and understood the information sheet for the above project and 
have had the opportunity to ask questions. 


2. |understand that during the discussion group the conversation will be recorded. | | 
3. |understand the recordings will only be accessed by researchers working on this project [| 
4. |understand that things my child say will be treated with the strictest confidence and my 


child’s name will not be used in any reports arising from the research. 


5. | CONSENT to my child taking part in the Online Maths Tuition Project Discussion Group | | 
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To be completed by the child 
Are you happy to take part in the study? 


YES NO 


(please circle the one you agree with) 


If you put a circle around ‘NO’ or you don’t want to take part, don’t sign your name! 


Child’s SIQMAUUIES 2 ee beesdieencgddeeranwettgecanbahaetie pled an danas nd deaebdbantac ee sane eheges Today’s 
GAlC hi sdad cs acincdectecaderncesgiacnenceted 
ee 
an A 
h: nt - THIRO SPACE 

UNIVERSITY Of York oe Nesta. LEARNING 

Online Maths Tuition_Parent-Child Consent Form v1.1 19May15 
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Appendix P: Process Evaluation: Delivery Partner Information Sheet 


an 
UNIVERSITY York W Durham 


ee 
Nesta... <n 


Education Endowment Foundation 


| | Finding out how to support and improve 
children’s maths skills 


! The Online Maths Tuition Project: A research 


study 


Information for Intervention Delivery Partners 


OMT_DeliveryPartners_PiS v1.1 19May1iS 
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"What is research? Why is this being done? 
Research is a way we try to find out answers to questions. 
Maths is an important skill that everyone needs to have. 
Maths can be hard and some children need help to improve 
their maths skills . We are trying to find out if using an 
online maths tutor can help children with their maths. 


How does the research work? 

As a member of the intervention delivery team you helped 
identify or set up schools for the project. We would like to 
talk to you about the successes & challenges you have expe- 
rienced during the lifetime of this project. We are interest- 
ed to hear your views including, the impact it has had on 
workloads, educating schools how to use the intervention, 


problems encountered with schools, methods of managing 
schools for example. To do this we would like to have a one 
to one discussion with you. 


What will I have to do? 


If you decide to take part you will need to be willing to be 
interviewed by one of our researchers. We will record the 
interview and transcribe it afterwards. We anticipate the 
interview to last between 30— 60 minutes. 


Will anyone else be told about what I say? 
Everything you say will be kept confidential and 
anonymous. . 


Do I have to take part? 
You can decide if you want to take part. If you don't want to 
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take part you do not have to give us a reason. If you 
decide to take part now you can always change your mind later 
on. 


Did anyone check to see if this study is OK to do? 

Before any research study is allowed to happen it has to be 
checked by a group of people called a Research Ethics 
Committee. They make sure that anyone who takes part is kept 
safe and the research is fair. This study has been checked by 
2 committees, one at the University of York and one at 
Durham University. 


Who is running the study? 
The study is being run by the University of York and Durham 
University. 


What happens next? 
If you want to take part, please complete the consent form 
and return this to the researcher... 


Who can I speak to about the study? 
You can speak to or email either: 


Debbie Humphry who will be carrying out the interviews 
+: 07831 811490; e: D.V.Humphry@sussex.ac.uk 


Natasha Mitchell who works for the University of York 
+: 01904 321655; e: natasha.mitchell@york.ac.uk 


If you are unhappy about the project you can contact 
Gillian Hampden-Thompson who is leading it. 
+: 01273 872601; e: 6.Hampden- Thompson@sussex.ac.uk 
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Appendix Q: Process Evaluation: Delivery Partner Consent 
Form 


Delivery Partner Consent Form 


If you would like to take part in the Online Maths Tuition Project Delivery Partner Interviews, please 
complete the following information and return this form to the researcher. 
Please initial each box 


1. I confirm that | have read and understood the information sheet for the above project and 
have had the opportunity to ask questions. 


2. |understand that the interview will be recorded & transcribed. 


[| 


3. | understand the recordings & transcriptions will only be accessed by researchers working on 
this project 


4. | understand anything shared during the interview will be treated with the strictest 
confidence and my name will not be used in any reports arising from the research. 
reports, but will not identify me by name. 


6. l|understand that my participation is voluntary and that | am free to withdraw at any time 
without giving any reason. 


7. | CONSENT to taking part in the Online Maths Tuition Project Delivery Partner Interviews 


5. | understand that the information gathered will be used to write research articles and [| 


Name Date Signature 


Name of Researcher Date Signature of Researcher 
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Table 11: Coefficients and standard deviations used for effect size calculations 


Co-efficient Residual SD | School SD | ICC estimate 
relating to | (Random (Random 
intervention) effect) effect) 
Primary analysis 0.005 2.553 1.778 0.3266 
Secondary analyses 
Repeating primary analysis | 0.10 2.589 1.818 0.3302 
adjusting only for baseline 
KS2 maths for non-identified | -0.50 3.372 1.488 0.1630 
pupils 
KS2 reading scores _ for | 0.05 3.416 1.730 0.2042 
identified pupils 
KS2 reading scores for non- | -0.11 2.8818 0.9725 0.1022 
identified pupils 
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Appendix S: Statistical analysis plan 


York Trials Unit 


Department of Health Sciences 


Online Maths Tuition 


Independent Evaluation of the Third Space Learning’s Online Maths 
Tuition 


STATISTICAL ANALYSIS PLAN 


Final v1.4 
York Trials Unit Version date: 16 Sep 2015 
Department of Health Sciences Author(s): Hannah Buckley 
University of York Chief Investigator: David Torgerson 
York, YO10 5DD Chief Investigator: Carole Torgerson 


Trial Coordinator: Natasha Mitchell 
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Note: This analysis plan was written post-randomisation and prior to receipt of any outcome data. 


Changes from published protocol 


It was originally planned that teachers at all participating primary schools would be asked to identify 
10 pupils, plus 3 reserve pupils, who would benefit from receiving online maths tuition in year 6. Initial 
interest in the trial was higher than anticipated and so the number of places allocated to each school 
was reduced to 8 in order to allow all schools to participate, whilst working with a limit of 600 funded 
places (300 of which would be intervention places). Following lack of data return from interested 
schools and withdrawals from recruited schools (i.e. schools who had returned baseline data) it was 
necessary to increase the number of places offered to some schools in order to fill the funded places. 
The order in which schools returned the data to York Trials Unit (YTU) was used to decide which schools 
would be offered 2 additional places (to give a total of 10 funded places) and which would continue to 
be offered 8 places. Schools were randomised in three waves based on time of return. All schools 
which were randomised in the first wave were given two additional places; these were filled using the 
first two reserve pupils the teacher had identified (based on the order the reserves appeared in the 
baseline data provided by schools). Schools in the second wave were ranked from first returned (1) to 
last returned (n) with higher numbers indicating a later return. Using the ranked list, each wave two 
school was offered two additional places in turn using the same process as for wave one; this continued 


until all 600 funded places had been filled. 


The protocol states that key stage 1 (KS1) data will be used as the pre-test outcome. It was later 
decided that since key stage 2 (KS2) predicted scores would be more highly correlated with the 
outcome of KS2 score (hence providing more power and precision) these predicted levels and sub- 
levels would be used in place of the KS1 data and be provided by schools at baseline prior to 


randomisation. 


The inclusion criteria in the protocol for identification of pupils to receive the intervention were: 
e year 6 pupils in 2014/15 
e predicted to achieve level 3 or an insecure level 4 in maths by the end of KS2 (based on teacher 
assessments) 
As this is a pragmatic randomised controlled trial, the decision was made to include pupils predicted 
to achieve a level 4a in maths by the end of KS2, if the school did not have enough year 6 pupils to 


identify 8 pupils meeting the original inclusion criteria or if any pupil’s level 4a was not deemed stable. 
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Definition of terms 


Level 3 or fragile level 4: level 3c, level 3b, level 3a, level 4c or level 4b 

KS1: Key stage 1 

KS2: Key stage 2 

SAT: Standard assessment tests 

FSM: Free school meals 

ITT: Intention to treat 

TSL: Third Space Learning 

YTU: York Trials Unit 

Non-identified pupil: a pupil who was not selected as either an intervention or reserve pupil OR a pupil 
who was selected as a reserve pupil and who was not offered an intervention place prior to the 
intervention commencing in order to fill places 

Identified pupil: a pupil who was initially selected to receive the intervention should the school be 
randomised to the intervention arm OR a pupil who was selected as a reserve pupil but selected to fill 
additional intervention slots should the school be randomised to the intervention (selection must have 
occurred prior to the intervention commencing otherwise the pupil will be considered “non-identified” 


for ITT purposes). 


Trial Objectives 


This trial aims to investigate the effectiveness of the Third Space Learning (TSL) online maths tuition 


intervention on the mathematical skills of participating pupils as compared with ‘business as usual’. 


Primary objective 


The primary objective of this trial is to investigate the effectiveness of the TSL online maths tuition 
programme on the mathematics skills of identified year 6 pupils who are struggling with maths and 


who are attending a participating primary school. 


Secondary objectives 


e To assess the impact of the intervention on the mathematical skills of year 6 non-identified 
pupils (see Section 2 for definition of non-identified pupil) 
e To assess the effectiveness of the intervention on the English skills of pupils identified to 


receive the online maths tuition 
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e To assess the impact of the intervention on the English skills of year 6 non-identified pupils 
(see Section 2 for definition of non-identified pupil) 
e To assess the effectiveness of the intervention on the mathematical skills of the subgroup of 


identified pupils eligible for FSM 


Design 


This is a pragmatic cluster randomised controlled trial. Approximately 60 schools have been randomly 
allocated to be offered the intervention either in 2014/5 (intervention group) or in 2015/6 (acting as 
the control group during 2014/15). Teachers at all participating primary schools were asked to identify 
8 year 5 pupils, plus 3 reserve pupils towards the end of the academic year 2013/2014 who would 
benefit from receiving online tutoring in year 6. As detailed in Section 0 (changes from published 
protocol) for some schools two reserve pupils were used to increase the number of identified pupils 
to 10; in these cases reserves were not replaced. Teachers were encouraged to target pupils who are 
predicted to achieve KS2 level 3 or a borderline KS2 level 4 in maths at the end of year 6. Pupils with 
special educational needs (SEN) were eligible for inclusion; however pupils who have a statement for 
special needs were not eligible for the intervention. 

The identified pupils in the primary schools randomised to the intervention group received the 


intervention in year 6 during 2014/5. 


The trial has been designed, conducted and will be reported to CONSORT standards (Altman et al, 
2011) in order to minimise all potential threats to internal validity, such as selection bias and a range 
of post randomisation biases (Cook and Campbell, 1969; Shadish, Cook and Campbell, 2002; Torgerson 
and Torgerson, 2008). In this way, unbiased estimates of the impact of the intervention will be 


provided. 


Full details of the background and trial design can be found within the protocol (Torgerson, et al., 


2014). 


Sample Size 


Ina previous trial evaluating a one-to-one maths intervention (Every Child Counts (ECC) trial; Torgerson 
et al, 2011) among primary school children over a single term, an effect size of 0.33 of a standard 
deviation was observed for one-to-one tuition by a classroom teacher. For the current study, the 
intervention will be delivered over nearly three terms; therefore, we might expect a similar or higher 


estimate. Assuming an intra-cluster correlation of 0.19 (from the ECC trial) and a pre and post-test 
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correlation of 0.67 (from national data), approximately 44 schools with 440 children needed to be 
recruited. Allowing for an attrition rate of 15%, we needed around 50-52 schools in our study (i.e., 25 
or 26 schools receiving the intervention from September 2014) to detect a difference of 0.33 of an 
effect size with 80% power. 

Funding was available for 60 schools (600 places; 300 identified intervention and 300 identified for a 
wait list control). This increased the number of schools which would allow us to detect the effect size 


of 0.33 with 85% power whilst allowing for 15% attrition. 


Randomisation 


Once pupil baseline data were received, schools were allocated on a 1:1 basis to either receive the 
intervention in 2014/2015 (the intervention group) or to receive the intervention in 2015/2016 (the 
wait list control group). The allocation was undertaken via minimisation using minimPy (Saghaei & 


Saghaei, 2011) and was conducted in waves (see Section 1). 


Naive minimisation with base probability 1.0 (i.e. deterministic minimisation) was conducted using the 
following as factors: 


e Number of pupils on roll (2 levels; less than 348 pupils and more than 348 pupils) 
e Percentage of pupils eligible for FSM (2 levels; less than 29% and more than 29%) 


e Percentage of KS2 maths at L4 and above in 2012/2013 (2 levels; less than 87% and more than 
87%). 


Cut-off values for levels were chosen based on baseline summary statistics from wave one schools. 


Outcomes 


Key stage 2 (KS2) standard assessment tests (SATs) which are mandatory, national tests will be used 
as the outcome measures in this trial. As schools are required to conduct these tests, missing data 
levels are expected to be low and related to absence or missing papers. Long term outcomes can be 


collected through the national pupil database. 


The primary outcome will be the KS2 maths SAT fine marked score. KS2 English SAT fine marked score 
will be used as a secondary outcome. The SATs will be administered as routine within the summer 
term of the academic year 2014/2015; teachers do not have access to the test prior to administration 
and hence there is no potential risk of bias due to ‘teaching to the test’. The fine marked score can be 


related to KS2 levels and sub-levels through point score equivalences. As the boundaries for these 
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vary slightly each year due to variation in national performance, the related levels will be obtained 


from the gov.uk website once they are published. 


Data 


Baseline data 


Baseline data were collected via an Excel spreadsheet which was sent to schools via email. The schools 
were then required to complete three tabs providing information on the school, teacher and pupil 
levels respectively. These spreadsheets were returned to York Trials Unit via the University of York 
DropOff Service (https://dropoff.york.ac.uk). Once baseline data were returned, schools were deemed 


to have been recruited and were randomised as detailed in Section 0. 


Outcome data 


KS2 outcome data will be collected from the National Pupil Database (NPD). Revised data as released 


in Oct 2015 will be used in this analysis. 


Cleaning and formatting 


Before any analyses are conducted, data cleaning of the whole data set will be conducted. This will 
include range checks and examination for logical inconsistencies. The first and last 10 entries for any 


derived variables will also be checked and summary statistics will be produced. 


Baseline data will be checked upon receipt for completeness and to ensure that all selected pupils 


meet the eligibility criteria. 


The final dataset will be formatted to apply with the funding body’s requirements for data transfer 
following the latest guidance (contact the Fischer Family Trust or Education Endowment Foundation 


for details of where to find the most up-to-date guidance). 


Analysis 


Analysis will be conducted using the principles of intention to treat, meaning that all pupils will be 
analysed as belonging to the group to which their school was randomised irrespective of whether or 
not they actually received the intervention throughout. All identified pupils (see Section 2 for 


definition) will be included in relevant analyses regardless of whether or not they actually received the 
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intervention. Reserve pupils who were not given an intervention space prior to the intervention 
starting in order to fill places will be treated as non-identified throughout even if they later received 


the intervention. 


Statistical significance will be assessed at the 5% level unless otherwise stated. 95% confidence 
intervals will be provided as appropriate. Regression based methods of analysis will be used. Model 
diagnostics will be used to check model assumptions and transformations considered if they do not 


hold. 


Effect sizes will be presented relating to all analyses alongside 95% confidence intervals. Effect size is 


defined as: 
a) Bintervention 
7 AN ee A 
Os 


where Bintervention is the difference in mean score between the intervention and control groups and o¢ 
is the residual standard deviation. Numerical values used to calculate the effect sizes for each analysis 


will be presented in the appendix of the final report. 


Baseline data 


School characteristics will be summarised by trial arm using appropriate summary statistics as will pupil 
level baseline data for identified pupils. Summaries will be produced both as randomised and as 
included in the primary analysis. No formal statistical testing will be conducted. Teacher level data 


will also be summarised. 


Trial completion (CONSORT flow diagram) 


A CONSORT diagram will be produced to show the flow of schools and pupils through the trial. This 


will include the number of pupils opting out of the trial. 


Descriptive analyses 


Raw unadjusted outcome results will be summarised by arm and related effect sizes calculated as 


required by the funding body. 


An estimate of the intra-cluster correlation co-efficient (ICC) associated with school for the primary 


outcome of KS2 maths fine mark score will be presented alongside a 95% Cl: 
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i. using data from all pupils attending a participating primary school 
ii. only using data from those included in the primary analysis. 
The correlation between the primary outcome of KS2 maths score and predicted KS2 level will also be 


estimated. 


The lessons pupils were intended to be withdrawn from will be summarised and considered in 


sensitivity analyses. 


Topics selected will be presented as frequencies and proportions out of the total number of sessions. 


Audio status for sessions will be presented in a similar manner. 


The number of reserve pupils who began receiving the intervention prior to commencement will be 


summarised. 


Primary analysis 


The primary objective of this study is to investigate the effectiveness of the intervention on the 
mathematics skills of the identified pupils. The difference in maths attainment between identified 
pupils in the intervention group and those in the control group will be compared using a linear mixed 
model with fine mark KS2 maths SAT score as the response variable. Group allocation, FSM status, 
gender, month of birth, and predicted KS2 maths score as collected at baseline will be included as 
covariates in the model. Adjustment will be made for cluster randomisation through the inclusion of 


school as a random effect. 


Secondary analyses 


An analogous approach to the primary analysis will be employed to compare maths attainment of the 
intervention and control reserve and unidentified pupils to assess for any spill-over effects for the 


untreated pupils. 
An analogous approach to the primary analysis will be used to assess for difference between the 


intervention and control identified pupils in terms of the secondary outcome of KS2 English score. This 


analysis will be repeated using reserve and unidentified pupils in the control and intervention groups. 
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The effect of the intervention on identified pupils who are eligible for FSM will be assessed via the 
inclusion of an interaction between FSM status and allocation in a repetition of the primary analysis. 
Statistical significance will be set at the 10% level as this trial is not powered to detect interactions. 

The funder requires the primary analysis to be repeated using data only from pupils eligible for FSM as 
a subgroup analysis. This will be conducted and results interpreted in light of the reduction in power 


this will cause. 


Sensitivity analyses 


The primary analysis will be repeated twice with the inclusion of: 
i. an interaction term between allocated group and whether teaching occurred during or 
outside of school hours to investigate if any effect is linked to additional maths tuition. 
ii. maths class as an additional random effect to account for any potential teacher effect. 
(Please note that this analysis may not be possible if small numbers of pupils are removed 


from each class). 


Compliance 


For this intervention group compliance will be summarised by term and overall in terms of: 
e Number and proportion of all sessions where a pupil is on time, late (more than 5 minutes), 
absent and cancelling 
e Number and proportion of all sessions where student engagement is assessed ready to 
learn/not engaged/very focused/distracted by the tutor 
e The mean and standard deviation of the end time of the session (negative where session 
finished early and positive where session overruns) 
The timing of sessions for those absent or cancelling will be summarised as during or outside of school 


hours to investigate any potential link. 


Non-compliance will be summarised in terms of student attendance (attended/absent) with thresholds 
of 75% and 50% considered. In addition, the number and proportion of pupils attending 75% and 50% 
of sessions on time will be summarised. The impact of non-compliance (should this occur and be 
measured appropriately) will be assessed using Complier Average Causal Effect (CACE) analysis taking 


an instrumental variable approach. If compliance is low, a ML approach may be used. 
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Appendix T: Padlock rating 


Design 


Fair and clear quasi- 
experimental design for 
comparison (e.g. RDD) with 
appropriate analysis, or 
experimental design with minor 
concerns about validity 


Power 


MDES < 0.2 


Well-matched comparison (using 
propensity score matching, or 
similar) or experimental design 
with moderate concerns about 

validity 


Weakly matched comparison or 
experimental design with major 
flaws 


MDES < 0.3 


MDES < 0.5 
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Attrition 


11-20% 


21-30% 


31-40% 


Comparison group with poor or 
no matching (E.g. volunteer 
versus others) 


MDES < 0.6 


51-50% 


No comparator 
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MDES > 0.6 


<50% 


Adjust 


Adjustment 
for Balance 


[- ] 


Adjustment 
for threats 
to internal 

validity 


[- ] 


Rating 


5 a 


4a 


1a 


90 


0a 


Affordable Online Maths Tuition 


Appendix U: Cost rating 


Cost ratings are based on the approximate cost per pupil per year of implementing the intervention over 
three years. Cost ratings are awarded using the following criteria. 


Cost Description 
£ Very low: less than £80 per pupil per year. 
EEL Low: up to about £200 per pupil per year. 


EEE Moderate: up to about £700 per pupil per year. 
ELEE High: up to £1,200 per pupil per year. 
ELLEEE Very high: over £1,200 per pupil per year. 
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You may re-use this document/publication (not including logos) free of charge in any format or 
medium, under the terms of the Open Government Licence v2.0. 


To view this licence, visit www .nationalarchives.zov.uk/doc/open-government-icence/version/2 or 


email: psi@nationalarchives.gsi.gov.uk 


Where we have identified any third party copyright information you will need to obtain permission 
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